Re: [PR] Summit EU 2024 announcement bar [pulsar-site]
visortelle commented on PR #842: URL: https://github.com/apache/pulsar-site/pull/842#issuecomment-1996579976 cc @jak78 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Summit EU 2024 announcement bar [pulsar-site]
visortelle opened a new pull request, #842: URL: https://github.com/apache/pulsar-site/pull/842 https://github.com/apache/pulsar-site/assets/9302460/fa734f73-ce71-42d5-b200-37542e9cd6a2;> https://registration.socio.events/e/pulsarvirtualsummiteurope2024 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] Version conflicts with the protobuf inside the pulsar client [pulsar]
pqab commented on issue #22263: URL: https://github.com/apache/pulsar/issues/22263#issuecomment-1996578754  pulsar-client-all has the same issue, the problem is the protobuf library inside the pulasr client is overriding our dependency, is it possible to decouple the protobuf from the client jar? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar-site) 01/01: Summit EU 2024 announcement bar
This is an automated email from the ASF dual-hosted git repository.
visortelle pushed a commit to branch summit-eu-announcement-bar
in repository https://gitbox.apache.org/repos/asf/pulsar-site.git
commit 9303e703dd728b98f1fd04cbfc4302ac5485a5d7
Author: visortelle
AuthorDate: Thu Mar 14 09:51:53 2024 +0400
Summit EU 2024 announcement bar
---
docusaurus.config.js | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/docusaurus.config.js b/docusaurus.config.js
index 016b4f807d99..f8825ae10f4b 100644
--- a/docusaurus.config.js
+++ b/docusaurus.config.js
@@ -151,8 +151,8 @@ module.exports = {
announcementBar: {
id: "summit",
content: renderAnnouncementBar(
-"Submit your stories to Pulsar Summit EMEA! ",
-"https://sessionize.com/pulsar-virtual-summit-europe-2024;
+"Join Pulsar Virtual Summit Europe 2024 ️",
+"https://registration.socio.events/e/pulsarvirtualsummiteurope2024;
),
backgroundColor: "#282826",
textColor: "#fff",
(pulsar-site) branch summit-eu-announcement-bar created (now 9303e703dd72)
This is an automated email from the ASF dual-hosted git repository. visortelle pushed a change to branch summit-eu-announcement-bar in repository https://gitbox.apache.org/repos/asf/pulsar-site.git at 9303e703dd72 Summit EU 2024 announcement bar This branch includes the following new commits: new 9303e703dd72 Summit EU 2024 announcement bar The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(pulsar) branch master updated: [improve][pip] PIP-343: Use picocli instead of jcommander (#22181)
This is an automated email from the ASF dual-hosted git repository.
zixuan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 63c0b47d720 [improve][pip] PIP-343: Use picocli instead of jcommander
(#22181)
63c0b47d720 is described below
commit 63c0b47d720871364f40b2b09af6777b87e9e0c1
Author: Zixuan Liu
AuthorDate: Thu Mar 14 13:39:16 2024 +0800
[improve][pip] PIP-343: Use picocli instead of jcommander (#22181)
Signed-off-by: Zixuan Liu
---
pip/pip-343.md | 143 +
1 file changed, 143 insertions(+)
diff --git a/pip/pip-343.md b/pip/pip-343.md
new file mode 100644
index 000..85fc323cba6
--- /dev/null
+++ b/pip/pip-343.md
@@ -0,0 +1,143 @@
+# PIP-343: Use picocli instead of jcommander
+
+# Motivation
+
+We use the [jcommander](https://github.com/cbeust/jcommander) to build the CLI
tool, which is a good library, and is
+stable, but it misses modern CLI features likes autocompletion, flag/command
suggestion, native image, etc.
+
+These features are very important because there are many commands in the CLI,
but the jcommander doesn't give friendly
+hints when we use incorrect flags/commands, which makes the user experience
not very friendly.
+
+In modern times, the [picocli](https://github.com/remkop/picocli) supports
these features, which is a popular library.
+
+The following is some comparison between jcommander and picocli:
+
+- Error prompt:
+ ```
+ bin/pulsar-admin clusters update cluster-a -b
+
+ # jcommander
+ Need to provide just 1 parameter
+
+ # picocli
+ Unknown option: '-b'
+ ```
+
+- Command suggestion:
+ ```
+ bin/pulsar-admin cluste
+
+ # jcommander
+ Expected a command, got cluste
+
+ # picocli
+ Unmatched argument at index 0: 'cluste'
+ Did you mean: pulsar-admin clusters?
+ ```
+
+# Goals
+
+## In Scope
+
+Use the picocli instead of the jcommander in our CLI tool:
+
+- bin/pulsar
+- bin/pulsar-admin
+- bin/pulsar-client
+- bin/pulsar-shell
+- bin/pulsar-perf
+
+I'm sure this will greatly improve the user experience, and in the future we
can also consider using native images to
+reduce runtime, and improve the CLI document based on picocli.
+
+## Out Scope
+
+This PR simply replaces jcommander and does not introduce any enhancements.
+
+In the CLI, [autocomplete](https://picocli.info/autocomplete.html) is an
important feature, and after this PIP is
+complete I will make a new PIP to support this feature.
+
+# Detailed Design
+
+## Design & Implementation Details
+
+The jcommander and picocli have similar APIs, this will make the migration
task very simple.
+
+This is [utility argument
syntax](https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap12.html):
+
+```
+utility_name[-a][-b][-c option_argument]
+[-d|-e][-f[option_argument]][operand...]
+```
+
+1. Use `@Command` instead of `@Parameters` to define the class as a command:
+
+ ```java
+ @Command(name = "my-command", description = "Operations on persistent
topics")
+ public class MyCommand {
+
+ }
+ ```
+
+2. Use `@Option` instead of `@Parameter` to defined the option of command:
+
+ ```java
+ @Option(names = {"-r", "--role"})
+ private String role;
+ ```
+
+3. Use `@Parameters` to get the operand of command:
+
+ ```java
+ @Parameters(description = "persistent://tenant/namespace/topic", arity = "1")
+ private String topicName;
+ ```
+
+4. Migrate jcommander converter to picocli converter:
+
+ ```java
+ public class TimeUnitToMillisConverter implements ITypeConverter {
+@Override
+public Long convert(String value) throws Exception {
+ return
TimeUnit.SECONDS.toMillis(RelativeTimeUtil.parseRelativeTimeInSeconds(value));
+}
+ }
+ ```
+
+5. Add the picocli entrypoint:
+
+ ```java
+ @Command
+ public class MyCommand implements Callable {
+// Picocli entrypoint.
+@Override
+public Integer call() throws Exception {
+// TODO
+// run();
+return 0;
+}
+ }
+ ```
+
+The above is a common migration approach, and then we need to consider
pulsar-shell and custom command separately.
+
+- pulsar-shell
+
+ This is an interactive shell based on jline3 and jcommander, which includes
pulsar-admin and pulsar-client commands.
+ The jcommander does not provide autocompletion because we have implemented
it ourselves. In picocli, they
+ have
[picocli-shell-jline3](https://github.com/remkop/picocli/blob/main/picocli-shell-jline3)
to help us quickly build
+ the interactive shell.
+
+- custom command:
+
+ This is an extension of pulsar-admin, and the plugin's implementation does
not depend on jcommander. Since the bridge
+ is used, we only need to change the generator code based on picocli.
+
+# Backward & Forward Compatibility
+
+Fully compatible.
+
+# Links
+
+* Mailing List discussion thread:
Re: [PR] [improve][pip] PIP-343: Use picocli instead of jcommander [pulsar]
nodece merged PR #22181: URL: https://github.com/apache/pulsar/pull/22181 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] Version conflicts with the protobuf inside the pulsar client [pulsar]
dao-jun commented on issue #22263: URL: https://github.com/apache/pulsar/issues/22263#issuecomment-1996490553 ```xml org.apache.pulsar pulsar-client-all 3.0.3 ``` instead of ```xml org.apache.pulsar pulsar-client 3.0.3 org.apache.pulsar pulsar-client-admin 3.0.3 ``` pulsar-client-all will shade these deps. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] Version conflicts with the protobuf inside the pulsar client [pulsar]
dao-jun commented on issue #22263: URL: https://github.com/apache/pulsar/issues/22263#issuecomment-1996455069 Try to use pulsar-client-all -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch branch-2.11 updated: [improve][sec] Align some namespace level policy authorisation check (#21640) (#22248)
This is an automated email from the ASF dual-hosted git repository.
technoboy pushed a commit to branch branch-2.11
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-2.11 by this push:
new 44ae31a85ca [improve][sec] Align some namespace level policy
authorisation check (#21640) (#22248)
44ae31a85ca is described below
commit 44ae31a85ca05086d7c4cf1e60708c365a43d4b3
Author: Jiwei Guo
AuthorDate: Thu Mar 14 12:27:30 2024 +0800
[improve][sec] Align some namespace level policy authorisation check
(#21640) (#22248)
---
.../pulsar/broker/admin/impl/NamespacesBase.java | 30 +-
.../apache/pulsar/broker/admin/v2/Namespaces.java | 3 ++-
2 files changed, 20 insertions(+), 13 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java
index 610b799fba6..0a602a631c2 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java
@@ -1349,7 +1349,8 @@ public abstract class NamespacesBase extends
AdminResource {
protected CompletableFuture internalSetPublishRateAsync(PublishRate
maxPublishMessageRate) {
log.info("[{}] Set namespace publish-rate {}/{}", clientAppId(),
namespaceName, maxPublishMessageRate);
-return validateSuperUserAccessAsync().thenCompose(__ ->
updatePoliciesAsync(namespaceName, policies -> {
+return validateNamespacePolicyOperationAsync(namespaceName,
PolicyName.RATE, PolicyOperation.WRITE)
+.thenCompose(__ -> updatePoliciesAsync(namespaceName, policies
-> {
policies.publishMaxMessageRate.put(pulsar().getConfiguration().getClusterName(),
maxPublishMessageRate);
log.info("[{}] Successfully updated the publish_max_message_rate
for cluster on namespace {}",
clientAppId(), namespaceName);
@@ -1378,7 +1379,8 @@ public abstract class NamespacesBase extends
AdminResource {
protected CompletableFuture internalRemovePublishRateAsync() {
log.info("[{}] Remove namespace publish-rate {}/{}", clientAppId(),
namespaceName, topicName);
-return validateSuperUserAccessAsync().thenCompose(__ ->
updatePoliciesAsync(namespaceName, policies -> {
+return validateNamespacePolicyOperationAsync(namespaceName,
PolicyName.RATE, PolicyOperation.WRITE)
+.thenCompose(__ -> updatePoliciesAsync(namespaceName, policies
-> {
if (policies.publishMaxMessageRate != null) {
policies.publishMaxMessageRate.remove(pulsar().getConfiguration().getClusterName());
}
@@ -1417,7 +1419,8 @@ public abstract class NamespacesBase extends
AdminResource {
@SuppressWarnings("deprecation")
protected CompletableFuture
internalSetTopicDispatchRateAsync(DispatchRateImpl dispatchRate) {
log.info("[{}] Set namespace dispatch-rate {}/{}", clientAppId(),
namespaceName, dispatchRate);
-return validateSuperUserAccessAsync().thenCompose(__ ->
updatePoliciesAsync(namespaceName, policies -> {
+return validateNamespacePolicyOperationAsync(namespaceName,
PolicyName.RATE, PolicyOperation.WRITE)
+.thenCompose(__ -> updatePoliciesAsync(namespaceName, policies
-> {
policies.topicDispatchRate.put(pulsar().getConfiguration().getClusterName(),
dispatchRate);
policies.clusterDispatchRate.put(pulsar().getConfiguration().getClusterName(),
dispatchRate);
log.info("[{}] Successfully updated the dispatchRate for cluster
on namespace {}", clientAppId(),
@@ -1444,7 +1447,8 @@ public abstract class NamespacesBase extends
AdminResource {
}
protected CompletableFuture internalDeleteTopicDispatchRateAsync() {
-return validateSuperUserAccessAsync().thenCompose(__ ->
updatePoliciesAsync(namespaceName, policies -> {
+return validateNamespacePolicyOperationAsync(namespaceName,
PolicyName.RATE, PolicyOperation.WRITE)
+.thenCompose(__ -> updatePoliciesAsync(namespaceName, policies
-> {
policies.topicDispatchRate.remove(pulsar().getConfiguration().getClusterName());
policies.clusterDispatchRate.remove(pulsar().getConfiguration().getClusterName());
log.info("[{}] Successfully delete the dispatchRate for cluster on
namespace {}", clientAppId(),
@@ -1461,7 +1465,7 @@ public abstract class NamespacesBase extends
AdminResource {
}
protected CompletableFuture
internalSetSubscriptionDispatchRateAsync(DispatchRateImpl dispatchRate) {
-return validateSuperUserAccessAsync()
+return validateNamespacePolicyOperationAsync(namespaceName,
PolicyName.RATE, PolicyOperation.WRITE)
.thenCompose(__ ->
Re: [PR] [improve][sec][branch-2.11] Align some namespace level policy authorisation check (#21640) [pulsar]
Technoboy- merged PR #22248: URL: https://github.com/apache/pulsar/pull/22248 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix] [ml] Add entry fail due to race condition about add entry failed/timeout and switch ledger [pulsar]
dao-jun commented on code in PR #1:
URL: https://github.com/apache/pulsar/pull/1#discussion_r1524225706
##
pulsar-common/src/main/java/org/apache/pulsar/common/mutable/AtomicMutableBoolean.java:
##
@@ -0,0 +1,165 @@
+package org.apache.pulsar.common.mutable;
+
+import org.apache.commons.lang3.BooleanUtils;
+import org.apache.commons.lang3.mutable.MutableBoolean;
+
+public class AtomicMutableBoolean extends MutableBoolean {
Review Comment:
can we just pass null to OpAddEntry when `setAddEntryTimeoutSeconds ` is
disabled?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [fix] [ml] Add entry fail due to race condition about add entry failed/timeout and switch ledger [pulsar]
dao-jun commented on code in PR #1:
URL: https://github.com/apache/pulsar/pull/1#discussion_r1524223866
##
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java:
##
@@ -200,7 +210,7 @@ public void addComplete(int rc, final LedgerHandle lh, long
entryId, Object ctx)
lh == null ? -1 : lh.getId(), entryId, dataLength, rc);
}
-if (rc != BKException.Code.OK) {
+if (rc != BKException.Code.OK || timeoutTriggered.isTrue()) {
Review Comment:
Maybe some addOps just still in the executor queue, and even not executed,
it is reasonable to switch to a new ledger?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[PR] [feat][site] PIP-264 Add OpenTelemetry metrics instructions [pulsar-site]
dragosvictor opened a new pull request, #841: URL: https://github.com/apache/pulsar-site/pull/841 This PR adds doc for https://github.com/apache/pulsar/pull/22010 and https://github.com/apache/pulsar/pull/22058. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [improve] PIP 342: Support OpenTelemetry metrics in Pulsar client [pulsar]
merlimat commented on code in PR #22178:
URL: https://github.com/apache/pulsar/pull/22178#discussion_r1524151273
##
pip/pip-342 OTel client metrics support.md:
##
@@ -0,0 +1,167 @@
+# PIP 342: Support OpenTelemetry metrics in Pulsar client
+
+## Motivation
+
+Current support for metric instrumentation in Pulsar client is very limited
and poses a lot of
+issues for integrating the metrics into any telemetry system.
+
+We have 2 ways that metrics are exposed today:
+
+1. Printing logs every 1 minute: While this is ok as it comes out of the box,
it's very hard for
+ any application to get the data or use it in any meaningful way.
+2. `producer.getStats()` or `consumer.getStats()`: Calling these methods will
get access to
+ the rate of events in the last 1-minute interval. This is problematic
because out of the
+ box the metrics are not collected anywhere. One would have to start its own
thread to
+ periodically check these values and export them to some other system.
+
+Neither of these mechanism that we have today are sufficient to enable
application to easily
+export the telemetry data of Pulsar client SDK.
+
+## Goal
+
+Provide a good way for applications to retrieve and analyze the usage of
Pulsar client operation,
+in particular with respect to:
+
+1. Maximizing compatibility with existing telemetry systems
+2. Minimizing the effort required to export these metrics
+
+## Why OpenTelemetry?
+
+[OpenTelemetry](https://opentelemetry.io/) is quickly becoming the de-facto
standard API for metric and
+tracing instrumentation. In fact, as part of
[PIP-264](https://github.com/apache/pulsar/blob/master/pip/pip-264.md),
+we are already migrating the Pulsar server side metrics to use OpenTelemetry.
+
+For Pulsar client SDK, we need to provide a similar way for application
builder to quickly integrate and
+export Pulsar metrics.
+
+### Why exposing OpenTelemetry directly in Pulsar API
+
+When deciding how to expose the metrics exporter configuration there are
multiple options:
+
+1. Accept an `OpenTelemetry` object directly in Pulsar API
+2. Build a pluggable interface that describe all the Pulsar client SDK events
and allow application to
+ provide an implementation, perhaps providing an OpenTelemetry included
option.
+
+For this proposal, we are following the (1) option. Here are the reasons:
+
+1. In a way, OpenTelemetry can be compared to [SLF4J](https://www.slf4j.org/),
in the sense that it provides an API
+ on top of which different vendor can build multiple implementations.
Therefore, there is no need to create a new
+ Pulsar-specific interface
+2. OpenTelemetry has 2 main artifacts: API and SDK. For the context of Pulsar
client, we will only depend on its
+ API. Applications that are going to use OpenTelemetry, will include the
OTel SDK
+3. Providing a custom interface has several drawbacks:
+1. Applications need to update their implementations every time a new
metric is added in Pulsar SDK
+2. The surface of this plugin API can become quite big when there are
several metrics
+3. If we imagine an application that uses multiple libraries, like Pulsar
SDK, and each of these has its own
+ custom way to expose metrics, we can see the level of integration
burden that is pushed to application
+ developers
+4. It will always be easy to use OpenTelemetry to collect the metrics and
export them using a custom metrics API. There
+ are several examples of this in OpenTelemetry documentation.
+
+## Public API changes
+
+### Enabling OpenTelemetry
+
+When building a `PulsarClient` instance, it will be possible to pass an
`OpenTelemetry` object:
+
+```java
+interface ClientBuilder {
+// ...
+ClientBuilder openTelemetry(io.opentelemetry.api.OpenTelemetry
openTelemetry);
+}
+```
+
+The common usage for an application would be something like:
+
+```java
+// Creates a OpenTelemetry instance using environment variables to configure it
+OpenTelemetry otel = AutoConfiguredOpenTelemetrySdk.builder().build()
+.getOpenTelemetrySdk();
+
+PulsarClient client = PulsarClient.builder()
+.serviceUrl("pulsar://localhost:6650")
+.build();
+
+//
+```
+
+Even without passing the `OpenTelemetry` instance to Pulsar client SDK, an
application using the OpenTelemetry
+agent, will be able to instrument the Pulsar client automatically, because we
default to use `GlobalOpenTelemetry.get()`.
+
+### Deprecating the old stats methods
+
+The old way of collecting stats will be deprecated in phases:
+ 1. Pulsar 3.3 - Old metrics deprecated, still enabled by default
+ 2. Pulsar 3.4 - Old metrics disabled by default
+ 3. Pulsar 4.0 - Old metrics removed
+
+Methods to deprecate:
+
+```java
+interface ClientBuilder {
+// ...
+@Deprecated
+ClientBuilder statsInterval(long statsInterval, TimeUnit unit);
+}
+
+interface Producer {
+@Deprecated
+ProducerStats getStats();
+}
+
+interface Consumer {
+@Deprecated
+
Re: [PR] [improve] WIP: PIP-342: OTel client metrics support [pulsar]
codecov-commenter commented on PR #22179: URL: https://github.com/apache/pulsar/pull/22179#issuecomment-1996290772 ## [Codecov](https://app.codecov.io/gh/apache/pulsar/pull/22179?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report Attention: Patch coverage is `91.08434%` with `37 lines` in your changes are missing coverage. Please review. > Project coverage is 73.64%. Comparing base [(`bbc6224`)](https://app.codecov.io/gh/apache/pulsar/commit/bbc62245c5ddba1de4b1e7cee4ab49334bc36277?dropdown=coverage=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) to head [(`980bae3`)](https://app.codecov.io/gh/apache/pulsar/pull/22179?dropdown=coverage=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache). > Report is 48 commits behind head on master. Additional details and impacted files [](https://app.codecov.io/gh/apache/pulsar/pull/22179?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) ```diff @@ Coverage Diff @@ ## master #22179 +/- ## + Coverage 73.57% 73.64% +0.07% - Complexity3262432783 +159 Files 1877 1885 +8 Lines139502 140057 +555 Branches 1529915352 +53 + Hits 102638 103151 +513 - Misses2890828930 +22 - Partials 7956 7976 +20 ``` | [Flag](https://app.codecov.io/gh/apache/pulsar/pull/22179/flags?src=pr=flags_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Coverage Δ | | |---|---|---| | [inttests](https://app.codecov.io/gh/apache/pulsar/pull/22179/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `26.76% <60.96%> (+2.17%)` | :arrow_up: | | [systests](https://app.codecov.io/gh/apache/pulsar/pull/22179/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `24.62% <54.93%> (+0.30%)` | :arrow_up: | | [unittests](https://app.codecov.io/gh/apache/pulsar/pull/22179/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `72.89% <91.08%> (+0.05%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files](https://app.codecov.io/gh/apache/pulsar/pull/22179?dropdown=coverage=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Coverage Δ | | |---|---|---| | [...va/org/apache/pulsar/client/api/ConsumerStats.java](https://app.codecov.io/gh/apache/pulsar/pull/22179?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cHVsc2FyLWNsaWVudC1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3B1bHNhci9jbGllbnQvYXBpL0NvbnN1bWVyU3RhdHMuamF2YQ==) | `0.00% <ø> (ø)` | | | [...va/org/apache/pulsar/client/api/ProducerStats.java](https://app.codecov.io/gh/apache/pulsar/pull/22179?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cHVsc2FyLWNsaWVudC1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3B1bHNhci9jbGllbnQvYXBpL1Byb2R1Y2VyU3RhdHMuamF2YQ==) | `0.00% <ø> (ø)` | | | [.../pulsar/client/impl/BatchMessageContainerImpl.java](https://app.codecov.io/gh/apache/pulsar/pull/22179?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL0JhdGNoTWVzc2FnZUNvbnRhaW5lckltcGwuamF2YQ==) | `80.89% <100.00%> (ø)` | | | [...e/pulsar/client/impl/BinaryProtoLookupService.java](https://app.codecov.io/gh/apache/pulsar/pull/22179?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL0JpbmFyeVByb3RvTG9va3VwU2VydmljZS5qYXZh) | `84.36% <100.00%> (+1.82%)` | :arrow_up: | |
Re: [PR] [fix][ws] Check the validity of config before start websocket service [pulsar]
codecov-commenter commented on PR #22231: URL: https://github.com/apache/pulsar/pull/22231#issuecomment-1996287513 ## [Codecov](https://app.codecov.io/gh/apache/pulsar/pull/22231?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report All modified and coverable lines are covered by tests :white_check_mark: > Project coverage is 73.64%. Comparing base [(`bbc6224`)](https://app.codecov.io/gh/apache/pulsar/commit/bbc62245c5ddba1de4b1e7cee4ab49334bc36277?dropdown=coverage=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) to head [(`828eb90`)](https://app.codecov.io/gh/apache/pulsar/pull/22231?dropdown=coverage=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache). > Report is 48 commits behind head on master. Additional details and impacted files [](https://app.codecov.io/gh/apache/pulsar/pull/22231?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) ```diff @@ Coverage Diff @@ ## master #22231 +/- ## + Coverage 73.57% 73.64% +0.07% + Complexity3262432266 -358 Files 1877 1879 +2 Lines139502 139801 +299 Branches 1529915340 +41 + Hits 102638 102960 +322 + Misses2890828864 -44 - Partials 7956 7977 +21 ``` | [Flag](https://app.codecov.io/gh/apache/pulsar/pull/22231/flags?src=pr=flags_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Coverage Δ | | |---|---|---| | [inttests](https://app.codecov.io/gh/apache/pulsar/pull/22231/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `26.59% <38.46%> (+2.01%)` | :arrow_up: | | [systests](https://app.codecov.io/gh/apache/pulsar/pull/22231/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `24.30% <0.00%> (-0.02%)` | :arrow_down: | | [unittests](https://app.codecov.io/gh/apache/pulsar/pull/22231/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `72.91% <61.53%> (+0.07%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files](https://app.codecov.io/gh/apache/pulsar/pull/22231?dropdown=coverage=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Coverage Δ | | |---|---|---| | [...a/org/apache/pulsar/websocket/ProducerHandler.java](https://app.codecov.io/gh/apache/pulsar/pull/22231?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cHVsc2FyLXdlYnNvY2tldC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL3dlYnNvY2tldC9Qcm9kdWNlckhhbmRsZXIuamF2YQ==) | `63.15% <100.00%> (+0.70%)` | :arrow_up: | | [...sar/websocket/service/WebSocketServiceStarter.java](https://app.codecov.io/gh/apache/pulsar/pull/22231?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cHVsc2FyLXdlYnNvY2tldC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL3dlYnNvY2tldC9zZXJ2aWNlL1dlYlNvY2tldFNlcnZpY2VTdGFydGVyLmphdmE=) | `76.00% <100.00%> (+1.53%)` | :arrow_up: | ... and [75 files with indirect coverage changes](https://app.codecov.io/gh/apache/pulsar/pull/22231/indirect-changes?src=pr=tree-more_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][broker] Fix incorrect unack msk count when dup ack a message (#20990) [pulsar]
Technoboy- merged PR #3: URL: https://github.com/apache/pulsar/pull/3 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch branch-2.10 updated: [fix][broker] Fix incorrect unack msk count when dup ack a message (#20990) (#22223)
This is an automated email from the ASF dual-hosted git repository.
technoboy pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-2.10 by this push:
new 2c85dd13384 [fix][broker] Fix incorrect unack msk count when dup ack a
message (#20990) (#3)
2c85dd13384 is described below
commit 2c85dd13384ea7622d9bb0715d04cb5e3dc4ed95
Author: Zixuan Liu
AuthorDate: Thu Mar 14 10:24:36 2024 +0800
[fix][broker] Fix incorrect unack msk count when dup ack a message (#20990)
(#3)
---
.../org/apache/pulsar/broker/service/Consumer.java | 24 ++--
.../pulsar/broker/service/BrokerServiceTest.java | 26
.../client/impl/KeySharedSubscriptionTest.java | 136 +
3 files changed, 98 insertions(+), 88 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java
index 8924b750eb6..a7c06d0c85d 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java
@@ -444,6 +444,7 @@ public class Consumer {
private CompletableFuture individualAckNormal(CommandAck ack,
Map properties) {
List positionsAcked = new ArrayList<>();
long totalAckCount = 0;
+boolean individualAck = false;
for (int i = 0; i < ack.getMessageIdsCount(); i++) {
MessageIdData msgId = ack.getMessageIdAt(i);
PositionImpl position;
@@ -467,14 +468,18 @@ public class Consumer {
} else {
position = PositionImpl.get(msgId.getLedgerId(),
msgId.getEntryId());
ackedCount = getAckedCountForMsgIdNoAckSets(batchSize,
position, ackOwnerConsumer);
+individualAck = true;
}
-addAndGetUnAckedMsgs(ackOwnerConsumer, -(int) ackedCount);
-
+if (individualAck) {
+if (checkCanRemovePendingAcksAndHandle(position, msgId)) {
+addAndGetUnAckedMsgs(ackOwnerConsumer, -(int) ackedCount);
+}
+} else {
+addAndGetUnAckedMsgs(ackOwnerConsumer, -(int) ackedCount);
+}
positionsAcked.add(position);
-checkCanRemovePendingAcksAndHandle(position, msgId);
-
checkAckValidationError(ack, position);
totalAckCount += ackedCount;
@@ -636,10 +641,11 @@ public class Consumer {
}
}
-private void checkCanRemovePendingAcksAndHandle(PositionImpl position,
MessageIdData msgId) {
+private boolean checkCanRemovePendingAcksAndHandle(PositionImpl position,
MessageIdData msgId) {
if (Subscription.isIndividualAckMode(subType) &&
msgId.getAckSetsCount() == 0) {
-removePendingAcks(position);
+return removePendingAcks(position);
}
+return false;
}
private Consumer getAckOwnerConsumer(long ledgerId, long entryId) {
@@ -886,7 +892,7 @@ public class Consumer {
*
* @param position
*/
-private void removePendingAcks(PositionImpl position) {
+private boolean removePendingAcks(PositionImpl position) {
Consumer ackOwnedConsumer = null;
if (pendingAcks.get(position.getLedgerId(), position.getEntryId()) ==
null) {
for (Consumer consumer : subscription.getConsumers()) {
@@ -907,7 +913,7 @@ public class Consumer {
if (ackedPosition != null) {
if
(!ackOwnedConsumer.getPendingAcks().remove(position.getLedgerId(),
position.getEntryId())) {
// Message was already removed by the other consumer
-return;
+return false;
}
if (log.isDebugEnabled()) {
log.debug("[{}-{}] consumer {} received ack {}", topicName,
subscription, consumerId, position);
@@ -921,7 +927,9 @@ public class Consumer {
ackOwnedConsumer.blockedConsumerOnUnackedMsgs = false;
flowConsumerBlockedPermits(ackOwnedConsumer);
}
+return true;
}
+return false;
}
public ConcurrentLongLongPairHashMap getPendingAcks() {
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BrokerServiceTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BrokerServiceTest.java
index 1135e2c8ff4..34d1d15764d 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BrokerServiceTest.java
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BrokerServiceTest.java
@@ -1435,4 +1435,30 @@ public class BrokerServiceTest extends BrokerTestBase {
assertTrue(conf.isForceDeleteTenantAllowed());
});
}
+
+@Test
+public void testDuplicateAcknowledgement() throws Exception {
+
Re: [PR] [improve] WIP: PIP-342: OTel client metrics support [pulsar]
KevinLiLu commented on code in PR #22179:
URL: https://github.com/apache/pulsar/pull/22179#discussion_r1524118347
##
pulsar-client/src/main/java/org/apache/pulsar/client/impl/metrics/MetricsUtil.java:
##
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.client.impl.metrics;
+
+import com.google.common.collect.Lists;
+import io.opentelemetry.api.common.AttributeKey;
+import io.opentelemetry.api.common.Attributes;
+import io.opentelemetry.api.common.AttributesBuilder;
+import java.util.ArrayList;
+import java.util.List;
+import lombok.experimental.UtilityClass;
+import org.apache.pulsar.common.naming.TopicName;
+
+@UtilityClass
+public class MetricsUtil {
+
+// By default, advice to use namespace level aggregation only
+private static final List> DEFAULT_AGGREGATION_LABELS
= Lists.newArrayList(
+AttributeKey.stringKey("pulsar.tenant"),
Review Comment:
Would it make sense to put the attribute values in a public enum/list so
users can directly reference/see all possible values without having to read
through the code? This might be helpful if a user wants to customize which
attributes are used for aggregation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [improve] PIP 342: Support OpenTelemetry metrics in Pulsar client [pulsar]
KevinLiLu commented on code in PR #22178:
URL: https://github.com/apache/pulsar/pull/22178#discussion_r1524114034
##
pip/pip-342 OTel client metrics support.md:
##
@@ -0,0 +1,167 @@
+# PIP 342: Support OpenTelemetry metrics in Pulsar client
+
+## Motivation
+
+Current support for metric instrumentation in Pulsar client is very limited
and poses a lot of
+issues for integrating the metrics into any telemetry system.
+
+We have 2 ways that metrics are exposed today:
+
+1. Printing logs every 1 minute: While this is ok as it comes out of the box,
it's very hard for
+ any application to get the data or use it in any meaningful way.
+2. `producer.getStats()` or `consumer.getStats()`: Calling these methods will
get access to
+ the rate of events in the last 1-minute interval. This is problematic
because out of the
+ box the metrics are not collected anywhere. One would have to start its own
thread to
+ periodically check these values and export them to some other system.
+
+Neither of these mechanism that we have today are sufficient to enable
application to easily
+export the telemetry data of Pulsar client SDK.
+
+## Goal
+
+Provide a good way for applications to retrieve and analyze the usage of
Pulsar client operation,
+in particular with respect to:
+
+1. Maximizing compatibility with existing telemetry systems
+2. Minimizing the effort required to export these metrics
+
+## Why OpenTelemetry?
+
+[OpenTelemetry](https://opentelemetry.io/) is quickly becoming the de-facto
standard API for metric and
+tracing instrumentation. In fact, as part of
[PIP-264](https://github.com/apache/pulsar/blob/master/pip/pip-264.md),
+we are already migrating the Pulsar server side metrics to use OpenTelemetry.
+
+For Pulsar client SDK, we need to provide a similar way for application
builder to quickly integrate and
+export Pulsar metrics.
+
+### Why exposing OpenTelemetry directly in Pulsar API
+
+When deciding how to expose the metrics exporter configuration there are
multiple options:
+
+1. Accept an `OpenTelemetry` object directly in Pulsar API
+2. Build a pluggable interface that describe all the Pulsar client SDK events
and allow application to
+ provide an implementation, perhaps providing an OpenTelemetry included
option.
+
+For this proposal, we are following the (1) option. Here are the reasons:
+
+1. In a way, OpenTelemetry can be compared to [SLF4J](https://www.slf4j.org/),
in the sense that it provides an API
+ on top of which different vendor can build multiple implementations.
Therefore, there is no need to create a new
+ Pulsar-specific interface
+2. OpenTelemetry has 2 main artifacts: API and SDK. For the context of Pulsar
client, we will only depend on its
+ API. Applications that are going to use OpenTelemetry, will include the
OTel SDK
+3. Providing a custom interface has several drawbacks:
+1. Applications need to update their implementations every time a new
metric is added in Pulsar SDK
+2. The surface of this plugin API can become quite big when there are
several metrics
+3. If we imagine an application that uses multiple libraries, like Pulsar
SDK, and each of these has its own
+ custom way to expose metrics, we can see the level of integration
burden that is pushed to application
+ developers
+4. It will always be easy to use OpenTelemetry to collect the metrics and
export them using a custom metrics API. There
+ are several examples of this in OpenTelemetry documentation.
+
+## Public API changes
+
+### Enabling OpenTelemetry
+
+When building a `PulsarClient` instance, it will be possible to pass an
`OpenTelemetry` object:
+
+```java
+interface ClientBuilder {
+// ...
+ClientBuilder openTelemetry(io.opentelemetry.api.OpenTelemetry
openTelemetry);
+}
+```
+
+The common usage for an application would be something like:
+
+```java
+// Creates a OpenTelemetry instance using environment variables to configure it
+OpenTelemetry otel = AutoConfiguredOpenTelemetrySdk.builder().build()
+.getOpenTelemetrySdk();
+
+PulsarClient client = PulsarClient.builder()
+.serviceUrl("pulsar://localhost:6650")
+.build();
+
+//
+```
+
+Even without passing the `OpenTelemetry` instance to Pulsar client SDK, an
application using the OpenTelemetry
+agent, will be able to instrument the Pulsar client automatically, because we
default to use `GlobalOpenTelemetry.get()`.
+
+### Deprecating the old stats methods
+
+The old way of collecting stats will be deprecated in phases:
+ 1. Pulsar 3.3 - Old metrics deprecated, still enabled by default
+ 2. Pulsar 3.4 - Old metrics disabled by default
+ 3. Pulsar 4.0 - Old metrics removed
+
+Methods to deprecate:
+
+```java
+interface ClientBuilder {
+// ...
+@Deprecated
+ClientBuilder statsInterval(long statsInterval, TimeUnit unit);
+}
+
+interface Producer {
+@Deprecated
+ProducerStats getStats();
+}
+
+interface Consumer {
+@Deprecated
+
[I] [Bug] Version conflicts with the protobuf inside the pulsar client [pulsar]
pqab opened a new issue, #22263: URL: https://github.com/apache/pulsar/issues/22263 ### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Version Pulsar client & client admin 3.0.2 ### Minimal reproduce step We have grpc application which is using protobuf 3.25.x, however when we build the proto files, the protobuf packed inside the pulsar client which is using an old version override our dependencies, causing issues in the grpc runtime envrionment, even if we tried to exclude from the gradle, it doesn't works, because it built inside the client jar directly ### What did you expect to see? The protobuf library inside the pulsar client shouldn't override our dependencies ### What did you see instead? It took priority to load the library from the pulsar client ### Anything else? We have a workaround to use pulsar-client-original & pulsar-client-admin-original client instead ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix] [client] Unclear error message when creating a consumer with two same topics [pulsar]
Technoboy- closed pull request #22255: [fix] [client] Unclear error message when creating a consumer with two same topics URL: https://github.com/apache/pulsar/pull/22255 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [improve] PIP 342: Support OpenTelemetry metrics in Pulsar client [pulsar]
merlimat commented on code in PR #22178:
URL: https://github.com/apache/pulsar/pull/22178#discussion_r1524099464
##
pip/pip-342 OTel client metrics support.md:
##
@@ -0,0 +1,201 @@
+# PIP 342: Support OpenTelemetry metrics in Pulsar client
+
+## Motivation
+
+Current support for metric instrumentation in Pulsar client is very limited
and poses a lot of
+issues for integrating the metrics into any telemetry system.
+
+We have 2 ways that metrics are exposed today:
+
+1. Printing logs every 1 minute: While this is ok as it comes out of the box,
it's very hard for
+ any application to get the data or use it in any meaningful way.
+2. `producer.getStats()` or `consumer.getStats()`: Calling these methods will
get access to
+ the rate of events in the last 1-minute interval. This is problematic
because out of the
+ box the metrics are not collected anywhere. One would have to start its own
thread to
+ periodically check these values and export them to some other system.
+
+Neither of these mechanism that we have today are sufficient to enable
application to easily
+export the telemetry data of Pulsar client SDK.
+
+## Goal
+
+Provide a good way for applications to retrieve and analyze the usage of
Pulsar client operation,
+in particular with respect to:
+
+1. Maximizing compatibility with existing telemetry systems
+2. Minimizing the effort required to export these metrics
+
+## Why OpenTelemetry?
+
+[OpenTelemetry](https://opentelemetry.io/) is quickly becoming the de-facto
standard API for metric and
+tracing instrumentation. In fact, as part of
[PIP-264](https://github.com/apache/pulsar/blob/master/pip/pip-264.md),
+we are already migrating the Pulsar server side metrics to use OpenTelemetry.
+
+For Pulsar client SDK, we need to provide a similar way for application
builder to quickly integrate and
+export Pulsar metrics.
+
+### Why exposing OpenTelemetry directly in Pulsar API
+
+When deciding how to expose the metrics exporter configuration there are
multiple options:
+
+ 1. Accept an `OpenTelemetry` object directly in Pulsar API
+ 2. Build a pluggable interface that describe all the Pulsar client SDK events
and allow application to
+provide an implementation, perhaps providing an OpenTelemetry included
option.
+
+For this proposal, we are following the (1) option. Here are the reasons:
+
+ 1. In a way, OpenTelemetry can be compared to
[SLF4J](https://www.slf4j.org/), in the sense that it provides an API
+on top of which different vendor can build multiple implementations.
Therefore, there is no need to create a new
+Pulsar-specific interface
+ 2. OpenTelemetry has 2 main artifacts: API and SDK. For the context of Pulsar
client, we will only depend on its
+API. Applications that are going to use OpenTelemetry, will include the
OTel SDK
+ 3. Providing a custom interface has several drawbacks:
+ 1. Applications need to update their implementations every time a new
metric is added in Pulsar SDK
+ 2. The surface of this plugin API can become quite big when there are
several metrics
+ 3. If we imagine an application that uses multiple libraries, like Pulsar
SDK, and each of these has its own
+custom way to expose metrics, we can see the level of integration
burden that is pushed to application
+developers
+ 4. It will always be easy to use OpenTelemetry to collect the metrics and
export them using a custom metrics API. There
+are several examples of this in OpenTelemetry documentation.
+
+## Public API changes
+
+### Enabling OpenTelemetry
+
+When building a `PulsarClient` instance, it will be possible to pass an
`OpenTelemetry` object:
+
+```java
+interface ClientBuilder {
+// ...
+ClientBuilder openTelemetry(io.opentelemetry.api.OpenTelemetry
openTelemetry);
+
+ClientBuilder openTelemetryMetricsCardinality(MetricsCardinality
metricsCardinality);
+}
+```
+
+The common usage for an application would be something like:
+
+```java
+// Creates a OpenTelemetry instance using environment variables to configure it
+OpenTelemetry otel=AutoConfiguredOpenTelemetrySdk.builder()
+.build().getOpenTelemetrySdk();
+
+PulsarClient client=PulsarClient.builder()
+.serviceUrl("pulsar://localhost:6650")
+.build();
+
+//
+```
+
+Cardinality enum will allow to select a default cardinality label to be
attached to the
+metrics:
+
+```java
+public enum MetricsCardinality {
+/**
+ * Do not add additional labels to metrics
+ */
+None,
+
+/**
+ * Label metrics by tenant
+ */
+Tenant,
+
+/**
+ * Label metrics by tenant and namespace
+ */
+Namespace,
+
+/**
+ * Label metrics by topic
+ */
+Topic,
+
+/**
+ * Label metrics by each partition
+ */
+Partition,
+}
+```
+
+The labels are addictive. For example, selecting `Topic` level would mean that
the metrics will be
+labeled like:
+
+```
Re: [PR] [improve] PIP 342: Support OpenTelemetry metrics in Pulsar client [pulsar]
merlimat commented on code in PR #22178: URL: https://github.com/apache/pulsar/pull/22178#discussion_r1524096177 ## pip/pip-342 OTel client metrics support.md: ## @@ -166,36 +122,51 @@ this is the proposed initial set of metrics to export. Additional metrics could be added later on, though it's better to start with the set of most important metrics and then evaluate any missing information. -| OTel metric name| Type | Unit| Description | -|-|---|-|| -| `pulsar.client.connections.opened` | Counter | connections | Counter of connections opened | -| `pulsar.client.connections.closed` | Counter | connections | Counter of connections closed | -| `pulsar.client.connections.failed` | Counter | connections | Counter of connections establishment failures | -| `pulsar.client.session.opened` | Counter | sessions| Counter of sessions opened. `type="producer"` or `consumer` | -| `pulsar.client.session.closed` | Counter | sessions| Counter of sessions closed. `type="producer"` or `consumer` | -| `pulsar.client.received`| Counter | messages| Number of messages received | -| `pulsar.client.received`| Counter | bytes | Number of bytes received | -| `pulsar.client.consumer.preteched.messages` | Gauge | messages| Number of messages currently sitting in the consumer pre-fetch queue | -| `pulsar.client.consumer.preteched` | Gauge | bytes | Total number of bytes currently sitting in the consumer pre-fetch queue | -| `pulsar.client.consumer.ack`| Counter | messages| Number of ack operations | -| `pulsar.client.consumer.nack` | Counter | messages| Number of negative ack operations | -| `pulsar.client.consumer.dlq`| Counter | messages| Number of messages sent to DLQ | -| `pulsar.client.consumer.ack.timeout`| Counter | messages| Number of ack timeouts events | -| `pulsar.client.producer.latency`| Histogram | seconds | Publish latency experienced by the application, includes client batching time | -| `pulsar.client.producer.rpc.latency`| Histogram | seconds | Publish RPC latency experienced internally by the client when sending data to receiving an ack | -| `pulsar.client.producer.published` | Counter | bytes | Bytes published | -| `pulsar.client.producer.pending.messages.count` | Gauge | messages| Pending messages for this producer | -| `pulsar.client.producer.pending.count` | Gauge | bytes | Pending bytes for this producer | +| OTel metric name| Type | Unit| Description | +|-|---|-|---| +| `pulsar.client.connection.opened` | Counter | connections | The number of connections opened | Review Comment: Added all the attributes here -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at:
[PR] [fix] Change the wrong `SourceInstanceStatusData` in `SinkInstanceStatus` [pulsar-client-go]
jiangpengcheng opened a new pull request, #1199: URL: https://github.com/apache/pulsar-client-go/pull/1199 *(If this PR fixes a github issue, please add `Fixes #`.)* Fixes # *(or if this PR is one task of a github issue, please add `Master Issue: #` to link to the master issue.)* Master Issue: # ### Motivation *Explain here the context, and why you're making that change. What is the problem you're trying to solve.* ### Modifications *Describe the modifications you've done.* ### Verifying this change - [ ] Make sure that the change passes the CI checks. *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Does this pull request potentially affect one of the following parts: *If `yes` was chosen, please highlight the changes* - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API: (yes / no) - The schema: (yes / no / don't know) - The default values of configurations: (yes / no) - The wire protocol: (yes / no) ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / GoDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a followup issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar-site) branch main updated: Docs sync done from apache/pulsar (#20c2f75)
This is an automated email from the ASF dual-hosted git repository. urfree pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/pulsar-site.git The following commit(s) were added to refs/heads/main by this push: new 7dd58e6e0f7e Docs sync done from apache/pulsar (#20c2f75) 7dd58e6e0f7e is described below commit 7dd58e6e0f7e61e9bb883e938ea4163508ce23ae Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> AuthorDate: Thu Mar 14 01:28:02 2024 + Docs sync done from apache/pulsar (#20c2f75) --- static/reference/next/pulsar-admin/sinks.md | 3 +++ static/reference/next/pulsar-admin/sources.md | 3 +++ static/swagger/master/swaggersink.json| 3 +++ static/swagger/master/swaggersource.json | 3 +++ static/swagger/master/v3/swaggersink.json | 3 +++ static/swagger/master/v3/swaggersource.json | 3 +++ 6 files changed, 18 insertions(+) diff --git a/static/reference/next/pulsar-admin/sinks.md b/static/reference/next/pulsar-admin/sinks.md index 60ad6ae96b52..194844462a45 100644 --- a/static/reference/next/pulsar-admin/sinks.md +++ b/static/reference/next/pulsar-admin/sinks.md @@ -54,6 +54,7 @@ $ pulsar-admin sinks create options | `--subs-name` | Pulsar source subscription name if user wants a specific subscription-name for input-topic consumer|null|| | `--auto-ack` | Whether or not the framework will automatically acknowledge messages|null|| | `--max-redeliver-count` | Maximum number of times that a message will be redelivered before being sent to the dead letter queue|null|| +| `--log-topic` | The topic to which the logs of a Pulsar Sink are produced|null|| | `--cleanup-subscription` | Whether delete the subscription when sink is deleted|null|| | `-a, --archive` | Path to the archive file for the sink. It also supports url-path [http/https/file (file protocol assumes that file already exists on worker host)] from which worker can download the package.|null|| @@ -104,6 +105,7 @@ $ pulsar-admin sinks update options | `--subs-name` | Pulsar source subscription name if user wants a specific subscription-name for input-topic consumer|null|| | `--auto-ack` | Whether or not the framework will automatically acknowledge messages|null|| | `--max-redeliver-count` | Maximum number of times that a message will be redelivered before being sent to the dead letter queue|null|| +| `--log-topic` | The topic to which the logs of a Pulsar Sink are produced|null|| | `--cleanup-subscription` | Whether delete the subscription when sink is deleted|null|| | `-a, --archive` | Path to the archive file for the sink. It also supports url-path [http/https/file (file protocol assumes that file already exists on worker host)] from which worker can download the package.|null|| @@ -301,6 +303,7 @@ $ pulsar-admin sinks localrun options | `--tls-trust-cert-path` | tls trust cert file path|null|| | `--metrics-port-start` | The starting port range for metrics server|null|| | `--auto-ack` | Whether or not the framework will automatically acknowledge messages|null|| +| `--log-topic` | The topic to which the logs of a Pulsar Sink are produced|null|| | `--cleanup-subscription` | Whether delete the subscription when sink is deleted|null|| diff --git a/static/reference/next/pulsar-admin/sources.md b/static/reference/next/pulsar-admin/sources.md index 9aed802ffe81..6d26c3fa3172 100644 --- a/static/reference/next/pulsar-admin/sources.md +++ b/static/reference/next/pulsar-admin/sources.md @@ -28,6 +28,7 @@ $ pulsar-admin sources create options | `--producer-config` | The custom producer configuration (as a JSON string)|null|| | `--batch-builder` | BatchBuilder provides two types of batch construction methods, DEFAULT and KEY_BASED. The default value is: DEFAULT|null|| | `--custom-runtime-options` | A string that encodes options to customize the runtime, see docs for configured runtime for details|null|| +| `--log-topic` | The topic to which the logs of a Pulsar Sink are produced|null|| | `--source-config` | Source config key/values|null|| | `--ram` | The RAM (in bytes) that need to be allocated per source instance (applicable only to the process and Docker runtimes)|null|| | `--secrets` | The map of secretName to an object that encapsulates how the secret is fetched by the underlying secrets provider|null|| @@ -65,6 +66,7 @@ $ pulsar-admin sources update options | `--producer-config` | The custom producer configuration (as a JSON string)|null|| | `--batch-builder` | BatchBuilder provides two types of batch construction methods, DEFAULT and KEY_BASED. The default value is: DEFAULT|null|| | `--custom-runtime-options` | A string that encodes options to customize the runtime, see docs for configured runtime for details|null|| +| `--log-topic` | The topic to which the logs of a Pulsar Sink are produced|null|| | `--source-config` | Source config key/values|null|| | `--ram` | The RAM (in bytes) that need to
Re: [PR] [improve] WIP: PIP-342: OTel client metrics support [pulsar]
merlimat commented on code in PR #22179:
URL: https://github.com/apache/pulsar/pull/22179#discussion_r1524088873
##
pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java:
##
@@ -386,7 +402,29 @@ protected ConsumerImpl(PulsarClientImpl client, String
topic, ConsumerConfigurat
topicNameWithoutPartition = topicName.getPartitionedTopicName();
+InstrumentProvider ip = client.instrumentProvider();
+consumersOpenedCounter =
ip.newCounter("pulsar.client.consumer.opened", Unit.Sessions,
+"Counter of sessions opened", topic, Attributes.empty());
Review Comment:
Good point. I've added `pulsar.subscription` attribute
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Bump google.golang.org/protobuf from 1.32.0 to 1.33.0 in /pulsar-function-go/examples [pulsar]
github-actions[bot] commented on PR #22262: URL: https://github.com/apache/pulsar/pull/22262#issuecomment-1996076446 @dependabot[bot] Please add the following content to your PR description and select a checkbox: ``` - [ ] `doc` - [ ] `doc-required` - [ ] `doc-not-needed` - [ ] `doc-complete` ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch dependabot/go_modules/pulsar-function-go/examples/google.golang.org/protobuf-1.33.0 created (now 0459fe2905c)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/pulsar-function-go/examples/google.golang.org/protobuf-1.33.0 in repository https://gitbox.apache.org/repos/asf/pulsar.git at 0459fe2905c Bump google.golang.org/protobuf in /pulsar-function-go/examples No new revisions were added by this update.
[PR] Bump google.golang.org/protobuf from 1.32.0 to 1.33.0 in /pulsar-function-go/examples [pulsar]
dependabot[bot] opened a new pull request, #22262: URL: https://github.com/apache/pulsar/pull/22262 Bumps google.golang.org/protobuf from 1.32.0 to 1.33.0. [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/pulsar/network/alerts). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] chore(deps): bump google.golang.org/protobuf from 1.30.0 to 1.33.0 [pulsar-client-go]
dependabot[bot] opened a new pull request, #1198: URL: https://github.com/apache/pulsar-client-go/pull/1198 Bumps google.golang.org/protobuf from 1.30.0 to 1.33.0. [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/pulsar-client-go/network/alerts). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar-client-go) branch dependabot/go_modules/google.golang.org/protobuf-1.33.0 created (now 9023d880)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/google.golang.org/protobuf-1.33.0 in repository https://gitbox.apache.org/repos/asf/pulsar-client-go.git at 9023d880 chore(deps): bump google.golang.org/protobuf from 1.30.0 to 1.33.0 No new revisions were added by this update.
Re: [PR] Bump google.golang.org/protobuf from 1.32.0 to 1.33.0 in /pulsar-function-go [pulsar]
github-actions[bot] commented on PR #22261: URL: https://github.com/apache/pulsar/pull/22261#issuecomment-1995819945 @dependabot[bot] Please add the following content to your PR description and select a checkbox: ``` - [ ] `doc` - [ ] `doc-required` - [ ] `doc-not-needed` - [ ] `doc-complete` ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch dependabot/go_modules/pulsar-function-go/google.golang.org/protobuf-1.33.0 created (now a137c475450)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/pulsar-function-go/google.golang.org/protobuf-1.33.0 in repository https://gitbox.apache.org/repos/asf/pulsar.git at a137c475450 Bump google.golang.org/protobuf in /pulsar-function-go No new revisions were added by this update.
[PR] Bump google.golang.org/protobuf from 1.32.0 to 1.33.0 in /pulsar-function-go [pulsar]
dependabot[bot] opened a new pull request, #22261: URL: https://github.com/apache/pulsar/pull/22261 Bumps google.golang.org/protobuf from 1.32.0 to 1.33.0. [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/pulsar/network/alerts). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix] flaky test: BrokerServiceTest.testBrokerStatsTopicLoadFailed [pulsar]
merlimat commented on code in PR #22256:
URL: https://github.com/apache/pulsar/pull/22256#discussion_r1523883392
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java:
##
@@ -1637,10 +1639,9 @@ private void createPersistentTopic(final String topic,
boolean createIfMissing,
Map properties,
@Nullable TopicPolicies topicPolicies) {
TopicName topicName = TopicName.get(topic);
final long topicCreateTimeMs =
TimeUnit.NANOSECONDS.toMillis(System.nanoTime());
-
topicFuture.exceptionally(t -> {
pulsarStats.recordTopicLoadFailed();
-return null;
+return Optional.empty();
Review Comment:
This change is not necessary since the stage returned by `exceptionally()`
is not being used
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java:
##
@@ -1235,22 +1235,24 @@ private CompletableFuture>
createNonPersistentTopic(String topic
CompletableFuture> topicFuture = new
CompletableFuture<>();
topicFuture.exceptionally(t -> {
pulsarStats.recordTopicLoadFailed();
-return null;
+return Optional.empty();
Review Comment:
Same as the other: This change is not necessary since the stage returned by
exceptionally() is not being used
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
(pulsar) branch master updated: [fix][fn] fix broken function-go test (#22260)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new 20c2f753527 [fix][fn] fix broken function-go test (#22260) 20c2f753527 is described below commit 20c2f75352792d07ca4435168542a3a6aef74c6f Author: Paul Gier AuthorDate: Wed Mar 13 15:35:37 2024 -0500 [fix][fn] fix broken function-go test (#22260) --- pulsar-function-go/examples/go.mod | 8 pulsar-function-go/examples/go.sum | 30 ++ pulsar-function-go/go.mod | 11 ++- pulsar-function-go/go.sum | 36 +--- 4 files changed, 41 insertions(+), 44 deletions(-) diff --git a/pulsar-function-go/examples/go.mod b/pulsar-function-go/examples/go.mod index dfc60e36522..f3e4bbca1e1 100644 --- a/pulsar-function-go/examples/go.mod +++ b/pulsar-function-go/examples/go.mod @@ -28,16 +28,16 @@ require ( github.com/klauspost/compress v1.10.8 // indirect github.com/konsorten/go-windows-terminal-sequences v1.0.3 // indirect github.com/linkedin/goavro/v2 v2.9.8 // indirect - github.com/matttproud/golang_protobuf_extensions v1.0.1 // indirect + github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect github.com/mitchellh/go-homedir v1.1.0 // indirect github.com/mtibben/percent v0.2.1 // indirect github.com/pierrec/lz4 v2.0.5+incompatible // indirect github.com/pkg/errors v0.9.1 // indirect github.com/pmezard/go-difflib v1.0.0 // indirect - github.com/prometheus/client_golang v1.12.2 // indirect + github.com/prometheus/client_golang v1.15.1 // indirect github.com/prometheus/client_model v0.4.0 // indirect - github.com/prometheus/common v0.32.1 // indirect - github.com/prometheus/procfs v0.7.3 // indirect + github.com/prometheus/common v0.42.0 // indirect + github.com/prometheus/procfs v0.9.0 // indirect github.com/sirupsen/logrus v1.6.0 // indirect github.com/spaolacci/murmur3 v1.1.0 // indirect github.com/stretchr/testify v1.8.4 // indirect diff --git a/pulsar-function-go/examples/go.sum b/pulsar-function-go/examples/go.sum index 3fabd79f802..46f02744115 100644 --- a/pulsar-function-go/examples/go.sum +++ b/pulsar-function-go/examples/go.sum @@ -72,7 +72,6 @@ github.com/bketelsen/crypt v0.0.4/go.mod h1:aI6NrJ0pMGgvZKL1iVgXLnfIFJtfV+bKCoqO github.com/bmizerany/perks v0.0.0-20141205001514-d9a9656a3a4b/go.mod h1:ac9efd0D1fsDb3EJvhqgXRbFx7bs2wqZ10HQPeU8U/Q= github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU= github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= -github.com/cespare/xxhash/v2 v2.1.2/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= github.com/cespare/xxhash/v2 v2.2.0 h1:DC2CZ1Ep5Y4k3ZQ899DldepgrayRUGE6BBZ/cd9Cj44= github.com/cespare/xxhash/v2 v2.2.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI= @@ -232,7 +231,6 @@ github.com/jpillora/backoff v1.0.0/go.mod h1:J/6gKK9jxlEcS3zixgDgUAsiuZ7yrSoa/FX github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU= github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4= github.com/json-iterator/go v1.1.11/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4= -github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU= github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk= github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU= @@ -251,6 +249,8 @@ github.com/kr/fs v0.1.0/go.mod h1:FFnZGqtBN9Gxj7eW1uZ42v5BccTP0vu6NEaFoC2HwRg= github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc= github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= github.com/kr/pretty v0.2.0/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI= +github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= +github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE= github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= @@ -259,8 +259,9 @@ github.com/linkedin/goavro/v2 v2.9.8/go.mod h1:UgQUb2N/pmueQYH9bfqFioWxzYCZXSfF8 github.com/magiconair/properties v1.8.5/go.mod
Re: [PR] [fix][fn] fix broken function-go test [pulsar]
merlimat merged PR #22260: URL: https://github.com/apache/pulsar/pull/22260 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][fn] fix broken function-go test [pulsar]
codecov-commenter commented on PR #22260: URL: https://github.com/apache/pulsar/pull/22260#issuecomment-1995496073 ## [Codecov](https://app.codecov.io/gh/apache/pulsar/pull/22260?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report All modified and coverable lines are covered by tests :white_check_mark: > Project coverage is 73.61%. Comparing base [(`bbc6224`)](https://app.codecov.io/gh/apache/pulsar/commit/bbc62245c5ddba1de4b1e7cee4ab49334bc36277?dropdown=coverage=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) to head [(`9c5cf0f`)](https://app.codecov.io/gh/apache/pulsar/pull/22260?dropdown=coverage=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache). > Report is 47 commits behind head on master. Additional details and impacted files [](https://app.codecov.io/gh/apache/pulsar/pull/22260?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) ```diff @@ Coverage Diff @@ ## master #22260 +/- ## + Coverage 73.57% 73.61% +0.03% - Complexity3262432755 +131 Files 1877 1879 +2 Lines139502 139794 +292 Branches 1529915340 +41 + Hits 102638 102906 +268 - Misses2890828909 +1 - Partials 7956 7979 +23 ``` | [Flag](https://app.codecov.io/gh/apache/pulsar/pull/22260/flags?src=pr=flags_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Coverage Δ | | |---|---|---| | [inttests](https://app.codecov.io/gh/apache/pulsar/pull/22260/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `26.48% <ø> (+1.90%)` | :arrow_up: | | [systests](https://app.codecov.io/gh/apache/pulsar/pull/22260/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `24.39% <ø> (+0.06%)` | :arrow_up: | | [unittests](https://app.codecov.io/gh/apache/pulsar/pull/22260/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `72.87% <ø> (+0.02%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more. [see 83 files with indirect coverage changes](https://app.codecov.io/gh/apache/pulsar/pull/22260/indirect-changes?src=pr=tree-more_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix] [ml] Add entry fail due to race condition about add entry failed/timeout and switch ledger [pulsar]
poorbarcode commented on code in PR #1:
URL: https://github.com/apache/pulsar/pull/1#discussion_r1523755969
##
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java:
##
@@ -200,7 +210,7 @@ public void addComplete(int rc, final LedgerHandle lh, long
entryId, Object ctx)
lh == null ? -1 : lh.getId(), entryId, dataLength, rc);
}
-if (rc != BKException.Code.OK) {
+if (rc != BKException.Code.OK || timeoutTriggered.isTrue()) {
Review Comment:
Yes, once a OpAddEntry called `handleAddFailure(lh)`, it will trigger a
ledger switch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [fix] [ml] Add entry fail due to race condition about add entry failed/timeout and switch ledger [pulsar]
poorbarcode commented on code in PR #1:
URL: https://github.com/apache/pulsar/pull/1#discussion_r1523755036
##
pulsar-common/src/main/java/org/apache/pulsar/common/mutable/AtomicMutableBoolean.java:
##
@@ -0,0 +1,165 @@
+package org.apache.pulsar.common.mutable;
+
+import org.apache.commons.lang3.BooleanUtils;
+import org.apache.commons.lang3.mutable.MutableBoolean;
+
+public class AtomicMutableBoolean extends MutableBoolean {
Review Comment:
Answered
[here](https://github.com/apache/pulsar/pull/1/files#r1523754758)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [fix] [ml] Add entry fail due to race condition about add entry failed/timeout and switch ledger [pulsar]
poorbarcode commented on code in PR #1: URL: https://github.com/apache/pulsar/pull/1#discussion_r1523754758 ## pulsar-common/src/main/java/org/apache/pulsar/common/mutable/AtomicMutableBoolean.java: ## Review Comment: When an OpAddEntry is completed, it will always call `atomicBoolean.isTrue()` at [this line](https://github.com/apache/pulsar/pull/1/files#diff-2c2fdb8cfc9daa4cd03ad15da3c7c7faeddee5ff3c102a56c8f33016bda82754R213). The value of Java's AtomicBoolean is modifier by `volatile`, it will reduce the performance. We can use a `MutableBoolean` to avoid using a `volatile` value when disabling the feature `setAddEntryTimeoutSeconds`. see the method `createTimeoutTriggeredMarker ` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] Can't split bundle: The size of split boundaries is not 1 - when using flow_or_qps_equally_divide [pulsar]
lhotari commented on issue #22254: URL: https://github.com/apache/pulsar/issues/22254#issuecomment-1994980423 @lordcheng10 any chance to check this issue? thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar-site) branch main updated: Update the footer "Security" link to point to the Security page
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/pulsar-site.git
The following commit(s) were added to refs/heads/main by this push:
new d694e3ffae7c Update the footer "Security" link to point to the
Security page
d694e3ffae7c is described below
commit d694e3ffae7c0f921f868302627f125b1b3d103b
Author: Lari Hotari
AuthorDate: Wed Mar 13 18:42:57 2024 +0200
Update the footer "Security" link to point to the Security page
---
docusaurus.config.js | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docusaurus.config.js b/docusaurus.config.js
index 15095e4760b7..016b4f807d99 100644
--- a/docusaurus.config.js
+++ b/docusaurus.config.js
@@ -325,7 +325,7 @@ module.exports = {
},
{
items: [
-{ label: "Security", href: "https://www.apache.org/security; },
+{ label: "Security", href: "https://pulsar.apache.org/security/; },
{
label: "Privacy",
href: "https://www.apache.org/foundation/policies/privacy.html;,
[PR] [fix][fn] fix broken function-go test [pulsar]
pgier opened a new pull request, #22260:
URL: https://github.com/apache/pulsar/pull/22260
Upgrade the prometheus client_golang library which changes the format of the
metric output to match the current tests.
### Motivation
A previous upgrade (https://github.com/apache/pulsar/pull/20579) to the
prometheus client library broke one of the tests because the spacing changes in
the serialized metric output. The specifc commit in prometheus that caused
this change is here: https://github.com/prometheus/client_golang/pull/955
Version v1.15.1 of the prometheus client lib changes the spacing back to
what it was in v1.11.0. However, it seems that this spacing is not consistent
between versions, so we may want to make the test more flexible in the future
to avoid hitting this issue again in future upgrades.
```
=== RUN TestExampleSummaryVec
stats_test.go:149:
Error Trace:
/Users/paulgier/projects/apache/pulsar/pulsar-function-go/pf/stats_test.go:149
Error: Not equal:
expected: "name:
\"pond_temperature_celsius\"\nhelp: \"The temperature of the frog
pond.\"\ntype: SUMMARY\nmetric: {\n label: {\nname: \"species\"\n
value: \"leiopelma-hochstetteri\"\n }\n summary: {\nsample_count: 0\n
sample_sum: 0\nquantile: {\n quantile: 0.5\n value: nan\n}\n
quantile: {\n quantile: 0.9\n value: nan\n}\nquantile: {\n
quantile: 0.99\n value: nan\n}\n }\n}\nmetric: {\n label: {\n
name: \"species\"\nvalue: \"lithobates-catesbeianus\"\n }\n summary: {\n
sample_count: 1000\nsample_sum: 31956.1017\nquantile: {\n
quantile: 0.5\n value: 32.4\n}\nquantile: {\n quantile:
0.9\n value: 41.4\n}\nquantile: {\n quantile: 0.99\n
value: 41.9\n}\n }\n}\nmetric: {\n label: {\nname: \"species\"\n
value: \"litoria-caerulea\"\n }\n summary: {\nsample_count: 1000\n
sample_sum: 29969.501\n
quantile: {\n quantile: 0.5\n value: 31.1\n}\nquantile:
{\n quantile: 0.9\n value: 41.3\n}\nquantile: {\n
quantile: 0.99\n value: 41.9\n}\n }\n}\n"
actual : "name:
\"pond_temperature_celsius\"\nhelp: \"The temperature of the frog
pond.\"\ntype: SUMMARY\nmetric: {\n label: {\nname: \"species\"\n
value: \"leiopelma-hochstetteri\"\n }\n summary: {\nsample_count: 0\n
sample_sum: 0\nquantile: {\n quantile: 0.5\n value: nan\n
}\nquantile: {\n quantile: 0.9\n value: nan\n}\n
quantile: {\n quantile: 0.99\n value: nan\n}\n }\n}\nmetric:
{\n label: {\nname: \"species\"\nvalue:
\"lithobates-catesbeianus\"\n }\n summary: {\nsample_count: 1000\n
sample_sum: 31956.1017\nquantile: {\n quantile: 0.5\n
value: 32.4\n}\nquantile: {\n quantile: 0.9\n value:
41.4\n}\nquantile: {\n quantile: 0.99\n value: 41.9\n
}\n }\n}\nmetric: {\n label: {\nname: \"species\"\nvalue:
\"litoria-caerulea\"\n }\n summary: {\nsample_count: 1
000\nsample_sum: 29969.501\nquantile: {\n quantile:
0.5\n value: 31.1\n}\nquantile: {\n quantile: 0.9\n
value: 41.3\n}\nquantile: {\n quantile: 0.99\n value:
41.9\n}\n }\n}\n"
```
### Modifications
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
*(Please pick either of the following options)*
This change is a trivial rework / code cleanup without any test coverage.
*(or)*
This change is already covered by existing tests, such as *(please describe
tests)*.
*(or)*
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads
(10MB)*
- *Extended integration test for recovery after broker failure*
### Does this pull request potentially affect one of the following parts:
*If the box was checked, please highlight the changes*
- [ ] Dependencies (add or upgrade a dependency)
- [ ] The public API
- [ ] The schema
- [ ] The default values of configurations
- [ ] The threading model
- [ ] The binary protocol
- [ ] The REST endpoints
- [ ] The admin CLI options
- [ X ] The metrics
- [ ] Anything that affects deployment
### Documentation
- [ ] `doc`
- [ ] `doc-required`
- [ X ] `doc-not-needed`
- [ ] `doc-complete`
### Matching PR in forked repository
PR in forked repository: https://github.com/pgier/pulsar/pull/12
--
This is an automated message from the Apache Git
Re: [PR] [improve][pip] PIP-339: Introducing the --log-topic Option for Pulsar Sinks and Sources [pulsar]
Technoboy- merged PR #22185: URL: https://github.com/apache/pulsar/pull/22185 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch master updated: [improve][pip] PIP-339: Introducing the --log-topic Option for Pulsar Sinks and Sources (#22185)
This is an automated email from the ASF dual-hosted git repository.
technoboy pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 392549cf926 [improve][pip] PIP-339: Introducing the --log-topic Option
for Pulsar Sinks and Sources (#22185)
392549cf926 is described below
commit 392549cf92683baebb3b0146cadc609ba394bbb3
Author: jiangpengcheng
AuthorDate: Wed Mar 13 23:52:15 2024 +0800
[improve][pip] PIP-339: Introducing the --log-topic Option for Pulsar Sinks
and Sources (#22185)
---
.../java/org/apache/pulsar/common/io/SinkConfig.java | 1 +
.../java/org/apache/pulsar/common/io/SourceConfig.java | 1 +
.../java/org/apache/pulsar/admin/cli/CmdSinks.java | 5 +
.../java/org/apache/pulsar/admin/cli/CmdSources.java | 5 +
.../apache/pulsar/functions/utils/SinkConfigUtils.java | 15 +++
.../pulsar/functions/utils/SourceConfigUtils.java | 15 +++
.../pulsar/functions/utils/SinkConfigUtilsTest.java| 18 ++
.../pulsar/functions/utils/SourceConfigUtilsTest.java | 17 +
8 files changed, 77 insertions(+)
diff --git
a/pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/io/SinkConfig.java
b/pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/io/SinkConfig.java
index 09b98249a4d..57e67c0bcee 100644
---
a/pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/io/SinkConfig.java
+++
b/pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/io/SinkConfig.java
@@ -94,4 +94,5 @@ public class SinkConfig {
private String transformFunction;
private String transformFunctionClassName;
private String transformFunctionConfig;
+private String logTopic;
}
diff --git
a/pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/io/SourceConfig.java
b/pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/io/SourceConfig.java
index 251e0bf810b..19919570457 100644
---
a/pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/io/SourceConfig.java
+++
b/pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/io/SourceConfig.java
@@ -72,4 +72,5 @@ public class SourceConfig {
private BatchSourceConfig batchSourceConfig;
// batchBuilder provides two types of batch construction methods, DEFAULT
and KEY_BASED
private String batchBuilder;
+private String logTopic;
}
diff --git
a/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdSinks.java
b/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdSinks.java
index 35dec576541..66b2816e777 100644
---
a/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdSinks.java
+++
b/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdSinks.java
@@ -410,6 +410,8 @@ public class CmdSinks extends CmdBase {
@Parameter(names = "--transform-function-config", description =
"Configuration of the transform function "
+ "applied before the Sink")
protected String transformFunctionConfig;
+@Parameter(names = "--log-topic", description = "The topic to which
the logs of a Pulsar Sink are produced")
+protected String logTopic;
protected SinkConfig sinkConfig;
@@ -605,6 +607,9 @@ public class CmdSinks extends CmdBase {
if (transformFunctionConfig != null) {
sinkConfig.setTransformFunctionConfig(transformFunctionConfig);
}
+if (null != logTopic) {
+sinkConfig.setLogTopic(logTopic);
+}
// check if configs are valid
validateSinkConfigs(sinkConfig);
diff --git
a/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdSources.java
b/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdSources.java
index ac6ff5e6845..c94fd49d717 100644
---
a/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdSources.java
+++
b/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdSources.java
@@ -365,6 +365,8 @@ public class CmdSources extends CmdBase {
@Parameter(names = "--secrets", description = "The map of secretName
to an object that encapsulates "
+ "how the secret is fetched by the underlying secrets
provider")
protected String secretsString;
+@Parameter(names = "--log-topic", description = "The topic to which
the logs of a Pulsar Sink are produced")
+protected String logTopic;
protected SourceConfig sourceConfig;
@@ -500,6 +502,9 @@ public class CmdSources extends CmdBase {
}
sourceConfig.setSecrets(secretsMap);
}
+if (null != logTopic) {
+sourceConfig.setLogTopic(logTopic);
+}
// check if source configs are valid
validateSourceConfigs(sourceConfig);
diff
Re: [PR] [improve][pip] PIP-336: WASM Support for pulsar-function-java [pulsar]
lburgazzoli commented on PR #21992: URL: https://github.com/apache/pulsar/pull/21992#issuecomment-1994627795 > In the description of [java-sdk](https://github.com/extism/java-sdk), JSON is also used for deserialization, and it requires the installation of additional dependencies, which is particularly uncomfortable. that is the old one, I new one will come based on chicory -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [improve][pip] PIP-336: WASM Support for pulsar-function-java [pulsar]
loongs-zhang commented on PR #21992: URL: https://github.com/apache/pulsar/pull/21992#issuecomment-1994618910 In the description of [java-sdk](https://github.com/extism/java-sdk), JSON is also used for deserialization, and it requires the installation of additional dependencies, which is particularly uncomfortable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch master updated: [improve][test] Reduce AdvertisedListenersMultiBrokerLeaderElectionTest flakiness (#22258)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 46c9ce9b068 [improve][test] Reduce
AdvertisedListenersMultiBrokerLeaderElectionTest flakiness (#22258)
46c9ce9b068 is described below
commit 46c9ce9b068a0e0191a9598e6c863bda3f428bc2
Author: Lari Hotari
AuthorDate: Wed Mar 13 07:48:35 2024 -0700
[improve][test] Reduce AdvertisedListenersMultiBrokerLeaderElectionTest
flakiness (#22258)
---
.../pulsar/broker/loadbalance/MultiBrokerLeaderElectionTest.java| 6 --
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/MultiBrokerLeaderElectionTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/MultiBrokerLeaderElectionTest.java
index f2712820d69..32f3acf4214 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/MultiBrokerLeaderElectionTest.java
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/MultiBrokerLeaderElectionTest.java
@@ -108,11 +108,12 @@ public class MultiBrokerLeaderElectionTest extends
MultiBrokerTestZKBaseTest {
List topicNames = IntStream.range(0, 500).mapToObj(i ->
topicNameBase + i)
.collect(Collectors.toList());
List allAdmins = getAllAdmins();
-@Cleanup("shutdown")
+@Cleanup("shutdownNow")
ExecutorService executorService =
Executors.newFixedThreadPool(allAdmins.size());
List>> resultFutures = new ArrayList<>();
// use Phaser to increase the chances of a race condition by
triggering all threads once
// they are waiting just before each lookupTopic call
+@Cleanup("forceTermination")
final Phaser phaser = new Phaser(1);
for (PulsarAdmin brokerAdmin : allAdmins) {
phaser.register();
@@ -149,11 +150,12 @@ public class MultiBrokerLeaderElectionTest extends
MultiBrokerTestZKBaseTest {
List topicNames = IntStream.range(0, 500).mapToObj(i ->
topicNameBase + i)
.collect(Collectors.toList());
List allClients = getAllClients();
-@Cleanup("shutdown")
+@Cleanup("shutdownNow")
ExecutorService executorService =
Executors.newFixedThreadPool(allClients.size());
List>> resultFutures = new ArrayList<>();
// use Phaser to increase the chances of a race condition by
triggering all threads once
// they are waiting just before each lookupTopic call
+@Cleanup("forceTermination")
final Phaser phaser = new Phaser(1);
for (PulsarClient brokerClient : allClients) {
phaser.register();
Re: [PR] [improve][test] Reduce AdvertisedListenersMultiBrokerLeaderElectionTest flakiness [pulsar]
lhotari merged PR #22258: URL: https://github.com/apache/pulsar/pull/22258 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [improve][pip] PIP-339: Introducing the --log-topic Option for Pulsar Sinks and Sources [pulsar]
codecov-commenter commented on PR #22185: URL: https://github.com/apache/pulsar/pull/22185#issuecomment-1994518303 ## [Codecov](https://app.codecov.io/gh/apache/pulsar/pull/22185?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report Attention: Patch coverage is `7.69231%` with `24 lines` in your changes are missing coverage. Please review. > Project coverage is 38.83%. Comparing base [(`bbc6224`)](https://app.codecov.io/gh/apache/pulsar/commit/bbc62245c5ddba1de4b1e7cee4ab49334bc36277?dropdown=coverage=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) to head [(`3725ab1`)](https://app.codecov.io/gh/apache/pulsar/pull/22185?dropdown=coverage=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache). > Report is 45 commits behind head on master. Additional details and impacted files [](https://app.codecov.io/gh/apache/pulsar/pull/22185?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) ```diff @@ Coverage Diff @@ ## master #22185 +/- ## = - Coverage 73.57% 38.83% -34.74% + Complexity3262412748-19876 = Files 1877 1739 -138 Lines139502 134534 -4968 Branches 1529914930 -369 = - Hits 10263852249-50389 - Misses2890875406+46498 + Partials 7956 6879 -1077 ``` | [Flag](https://app.codecov.io/gh/apache/pulsar/pull/22185/flags?src=pr=flags_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Coverage Δ | | |---|---|---| | [inttests](https://app.codecov.io/gh/apache/pulsar/pull/22185/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `26.46% <0.00%> (+1.87%)` | :arrow_up: | | [systests](https://app.codecov.io/gh/apache/pulsar/pull/22185/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `24.33% <7.69%> (+<0.01%)` | :arrow_up: | | [unittests](https://app.codecov.io/gh/apache/pulsar/pull/22185/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `32.74% <9.09%> (-40.10%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files](https://app.codecov.io/gh/apache/pulsar/pull/22185?dropdown=coverage=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Coverage Δ | | |---|---|---| | [...n/java/org/apache/pulsar/common/io/SinkConfig.java](https://app.codecov.io/gh/apache/pulsar/pull/22185?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cHVsc2FyLWNsaWVudC1hZG1pbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3B1bHNhci9jb21tb24vaW8vU2lua0NvbmZpZy5qYXZh) | `91.89% <100.00%> (-2.56%)` | :arrow_down: | | [...java/org/apache/pulsar/common/io/SourceConfig.java](https://app.codecov.io/gh/apache/pulsar/pull/22185?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cHVsc2FyLWNsaWVudC1hZG1pbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3B1bHNhci9jb21tb24vaW8vU291cmNlQ29uZmlnLmphdmE=) | `87.50% <100.00%> (-8.16%)` | :arrow_down: | | [...ain/java/org/apache/pulsar/admin/cli/CmdSinks.java](https://app.codecov.io/gh/apache/pulsar/pull/22185?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cHVsc2FyLWNsaWVudC10b29scy9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2FkbWluL2NsaS9DbWRTaW5rcy5qYXZh) | `0.00% <0.00%> (-43.47%)` | :arrow_down: | | [...n/java/org/apache/pulsar/admin/cli/CmdSources.java](https://app.codecov.io/gh/apache/pulsar/pull/22185?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cHVsc2FyLWNsaWVudC10b29scy9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2FkbWluL2NsaS9DbWRTb3VyY2VzLmphdmE=) | `0.00% <0.00%> (-46.48%)` | :arrow_down: | |
Re: [PR] [feat] Add startPaused setting to consumer [pulsar-client-cpp]
BewareMyPower commented on code in PR #416:
URL: https://github.com/apache/pulsar-client-cpp/pull/416#discussion_r1523345472
##
tests/BasicEndToEndTest.cc:
##
@@ -968,6 +967,60 @@ TEST(BasicEndToEndTest, testMessageListenerPause) {
client.close();
}
+void testStartPaused(bool isPartitioned) {
+Client client(lookupUrl);
+std::string topicName =
+isPartitioned ? "testStartPausedWithPartitionedTopic" :
"testStartPausedWithNonPartitionedTopic";
+std::string subName = "sub";
+
+if (isPartitioned) {
+// Call admin api to make it partitioned
+std::string url = adminUrl + "admin/v2/persistent/public/default/" +
topicName + "/partitions";
+int res = makePutRequest(url, "5");
+LOG_INFO("res = " << res);
+ASSERT_FALSE(res != 204 && res != 409);
+}
+
+Producer producer;
+Result result = client.createProducer(topicName, producer);
+
+// Initializing global Count
+globalCount = 0;
+
+ConsumerConfiguration consumerConfig;
+consumerConfig.setMessageListener(
+std::bind(messageListenerFunction, std::placeholders::_1,
std::placeholders::_2));
+consumerConfig.setStartPaused(true);
+Consumer consumer;
+// Removing dangling subscription from previous test failures
+result = client.subscribe(topicName, subName, consumerConfig, consumer);
+consumer.unsubscribe();
+
+result = client.subscribe(topicName, subName, consumerConfig, consumer);
+ASSERT_EQ(ResultOk, result);
+
+int numOfMessages = 50;
+for (int i = 0; i < numOfMessages; i++) {
+std::string messageContent = "msg-" + std::to_string(i);
+Message msg = MessageBuilder().setContent(messageContent).build();
+ASSERT_EQ(ResultOk, producer.send(msg));
+}
+
+std::this_thread::sleep_for(std::chrono::microseconds(2 * 1000 * 1000));
+ASSERT_EQ(globalCount, 0);
+consumer.resumeMessageListener();
+std::this_thread::sleep_for(std::chrono::microseconds(2 * 1000 * 1000));
Review Comment:
Use `waitUntil` to avoid sleeping too long, see examples in
tests/ProducerTest.cc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [fix] flaky test: BrokerServiceTest.testBrokerStatsTopicLoadFailed [pulsar]
dao-jun commented on PR #22256: URL: https://github.com/apache/pulsar/pull/22256#issuecomment-1994507204 @lhotari @poorbarcode Could you plz take a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [improve][pip] PIP-339: Introducing the --log-topic Option for Pulsar Sinks and Sources [pulsar]
Technoboy- closed pull request #22185: [improve][pip] PIP-339: Introducing the --log-topic Option for Pulsar Sinks and Sources URL: https://github.com/apache/pulsar/pull/22185 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch master updated: [fix] [broker] Fix typo in CompactedTopicImpl. (#22235)
This is an automated email from the ASF dual-hosted git repository.
technoboy pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 55625d9ded1 [fix] [broker] Fix typo in CompactedTopicImpl. (#22235)
55625d9ded1 is described below
commit 55625d9ded12a4750b201b66fca0c92a662f24bb
Author: thetumbled <52550727+thetumb...@users.noreply.github.com>
AuthorDate: Wed Mar 13 21:20:55 2024 +0800
[fix] [broker] Fix typo in CompactedTopicImpl. (#22235)
---
.../main/java/org/apache/pulsar/compaction/CompactedTopicImpl.java| 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactedTopicImpl.java
b/pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactedTopicImpl.java
index dfafbc41cb4..f67f28733bd 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactedTopicImpl.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactedTopicImpl.java
@@ -62,7 +62,7 @@ import org.slf4j.LoggerFactory;
public class CompactedTopicImpl implements CompactedTopic {
static final long NEWER_THAN_COMPACTED = -0xfeed0fbaL;
static final long COMPACT_LEDGER_EMPTY = -0xfeed0fbbL;
-static final int DEFAULT_STARTPOINT_CACHE_SIZE = 100;
+static final int DEFAULT_MAX_CACHE_SIZE = 100;
private final BookKeeper bk;
@@ -254,7 +254,7 @@ public class CompactedTopicImpl implements CompactedTopic {
}
}, null);
return promise.thenApply((ledger) -> new CompactedTopicContext(
- ledger, createCache(ledger,
DEFAULT_STARTPOINT_CACHE_SIZE)));
+ ledger, createCache(ledger,
DEFAULT_MAX_CACHE_SIZE)));
}
private static CompletableFuture tryDeleteCompactedLedger(BookKeeper
bk, long id) {
Re: [PR] [fix] [broker] Fix typo in CompactedTopicImpl. [pulsar]
Technoboy- merged PR #22235: URL: https://github.com/apache/pulsar/pull/22235 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [improve][broker] PIP-226: Add JWKS support for AuthenticationProviderToken [pulsar]
nodece commented on code in PR #22215:
URL: https://github.com/apache/pulsar/pull/22215#discussion_r1523246750
##
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authentication/AuthenticationProviderToken.java:
##
@@ -77,6 +90,9 @@ public class AuthenticationProviderToken implements
AuthenticationProvider {
// token validation.
static final String CONF_TOKEN_ALLOWED_CLOCK_SKEW_SECONDS =
"tokenAllowedClockSkewSeconds";
+// When JSON Web Key Set is configured
+static final String CONF_TOKEN_KEY_SET_KEY = "tokenKeySetKey";
Review Comment:
Good catch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [I] [Enhancement] authentication refresh support for mTLS auth [pulsar]
nodece commented on issue #22125: URL: https://github.com/apache/pulsar/issues/22125#issuecomment-1994378432 > @nodece - just to confirm the scope of what u are thinking. Would this certificate monitor _**JUST**_ be checking the certificate provided by the client to auth? Or should it **also** check the expiry of the certificate used to setup the TLS transport, i.e. the `tlsCertificateFilePath` cert. This is the same thing. Please note that the client never sends an auth certificate, which from `javax.net.ssl.SSLSession#getPeerCertificates`. Therefore, your client needs to reconnect to the broker for refresh the certificate. > if the client certificate has expired - the client needs to make a new TCP connection (which will of course negotiate TLS again also). This is why my PoC PR is not optimal(?) This is correct. > Do u have any further tips / advice re fixing this? Could this be implemented similar to how `refreshAuthenticationCredentials` is scheduled in `ServerCnx`? If i could get some direction I could help tackle this bug. You can refer to the `authRefreshTask` in the `ServerCnx`, new a `peerCertExpireCheckTask` schedule take with fixd delay(The expiration time of the certificate minus the current time), when the certificate expires, you can disconnect this connection. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [fix] Reconnection logic and Backoff policy doesn't work correctly [pulsar-client-go]
crossoverJie opened a new pull request, #1197: URL: https://github.com/apache/pulsar-client-go/pull/1197 Fixes #1187 ### Modifications - Move `backoff.go` to the `backoff` directory (because there are circular dependencies, they are not moved to the pulsar directory.) - Create a new method for `BackOffPolicy` interface `IsMaxBackoffReached(delayReconnectTime, totalDelayReconnectTime time.Duration) bool` This is a **breaking change** that modifies the package name and interface name. Package: `internal`->`backoff` Interface name: `BackoffPolicy`-> `Policy` ### Verifying this change - [x] Make sure that the change passes the CI checks. ### Does this pull request potentially affect one of the following parts: *If `yes` was chosen, please highlight the changes* - Dependencies (does it add or upgrade a dependency): (no) - The public API: (yes) - The schema: (no) - The default values of configurations: (no) - The wire protocol: (no) ### Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (GoDocs) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a followup issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [improve][broker] PIP-226: Add JWKS support for AuthenticationProviderToken [pulsar]
damienburke commented on code in PR #22215:
URL: https://github.com/apache/pulsar/pull/22215#discussion_r1523024891
##
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authentication/AuthenticationProviderToken.java:
##
@@ -77,6 +90,9 @@ public class AuthenticationProviderToken implements
AuthenticationProvider {
// token validation.
static final String CONF_TOKEN_ALLOWED_CLOCK_SKEW_SECONDS =
"tokenAllowedClockSkewSeconds";
+// When JSON Web Key Set is configured
+static final String CONF_TOKEN_KEY_SET_KEY = "tokenKeySetKey";
Review Comment:
should this be added to any of the .conf files, e.g. `conf/broker.conf`
similar question, but i assume there will be a separate ticket to update our
docs, e.g.
[this](https://pulsar.apache.org/reference/#/next/config/reference-configuration-broker)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [I] [Enhancement] authentication refresh support for mTLS auth [pulsar]
damienburke commented on issue #22125: URL: https://github.com/apache/pulsar/issues/22125#issuecomment-1994106847 Just also to restate some of what u said - just for my own understanding - if u can confirm if accurate @nodece? When the pulsar broker identifies that the authenticated credentials have expired - and it issues the challenge to the client to refresh their credentials (e.g. as supported by "token" auth method - and as triggered by `authenticationRefreshCheckSeconds`) - the client uses the **same** TCP connection, with the **same** TLS encryption to send their refreshed credentials. Whereas your point if that, if the client (or sever?) cert has expired - the client needs to make a new TCP connection (and negotiate TLS) again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Enhancement] authentication refresh support for mTLS auth [pulsar]
damienburke commented on issue #22125: URL: https://github.com/apache/pulsar/issues/22125#issuecomment-1994085867 @nodece - just to confirm the scope of what u are thinking. Would this certificate expire check JUST be checking the certificate used by the client to auth? Or should it also check the expiry of the certificate used to setup the TLS transport, i.e. the `tlsCertificateFilePath` cert. Do u have any further tips / advice re fixing this? Could this be implemented similar to how `refreshAuthenticationCredentials` is scheduled in `ServerCnx`? If i could get some direction I could help tackle this bug. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [D] Can I Use the Client of a Later Version to Connect to the Server of an Earlier Version? [pulsar]
GitHub user jak78 added a comment to the discussion: Can I Use the Client of a Later Version to Connect to the Server of an Earlier Version? Hi @ouyang-xtt! That should work. Quoting the [documentation](https://pulsar.apache.org/docs/3.2.x/client-libraries/): > A design goal of Pulsar is to ensure full compatibility between all versions > of the client and the broker. When a client connects to a broker they agree > upon a version of the protocol to use. As a result, new features that rely on > an updates to the protocol are only available when using both newer clients > and newer brokers. It's important to note that version 2.10.4 is outdated and contains security vulnerabilities. We recommend upgrading to either a Long-Term Support (LTS) version (3.0.x) or a feature version (3.2.x). If migration to the 3.x series isn't feasible in short term, prioritize upgrading to version 2.10.6 as soon as possible. Best, GitHub link: https://github.com/apache/pulsar/discussions/22259#discussioncomment-8771152 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
Re: [PR] [cleanup][ml] ManagedCursor clean up. [pulsar]
dao-jun commented on PR #22246: URL: https://github.com/apache/pulsar/pull/22246#issuecomment-1994002755 > @dao-jun The PR touched the critical path of Pulsar, we should wait for more reviewers to review the PR. If a PR get merged with one reviewer and merged in a few hours, the committers will lost the chance to review your PR and new potential risks might introduce to the Pulsar codebase. Now, so many companies are relaying on the stable release, even if this is a trivial changes to cleanup the managed cursor, give more time for other reviewer will still benefit Pulsar. WDYT? got it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Enhancement] authentication refresh support for mTLS auth [pulsar]
nodece commented on issue #22125: URL: https://github.com/apache/pulsar/issues/22125#issuecomment-1993919553 I think there are two things here: 1. The Pulsar broker/client support reloads the TLS certificate and key when running. This feature is supported. 2. What happens when the certificate expires. The connection didn't disconnect. This is a bug, we need to add a certificate monitor to check if the certificate expires. @damienburke's PR: https://github.com/apache/pulsar/compare/master...damienburke:pulsar:master This is a way to check if the certificate expires, which is not optimal, we also have TLS encrypted transport, not auth. I think we should check each TLS connection, once the certificate expires we need to disconnect the client. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch branch-3.1 updated: [fix][sec] Revert "[fix][sec] Add a check for the input time value (apache#22023)" (#22243)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new c0fd8d3647c [fix][sec] Revert "[fix][sec] Add a check for the input
time value (apache#22023)" (#22243)
c0fd8d3647c is described below
commit c0fd8d3647c77c83cfbe9fd18c3ee41eaad56976
Author: Xiangying Meng <55571188+liangyepianz...@users.noreply.github.com>
AuthorDate: Wed Mar 13 09:30:45 2024 +0800
[fix][sec] Revert "[fix][sec] Add a check for the input time value
(apache#22023)" (#22243)
(cherry picked from commit 639a1b73a34bbb5115503d4c7783bb2a2770)
---
.../main/java/org/apache/bookkeeper/mledger/ManagedLedgerConfig.java | 5 +
.../bookkeeper/mledger/impl/ManagedLedgerFactoryMBeanImpl.java | 2 --
.../org/apache/bookkeeper/mledger/impl/ManagedLedgerMBeanImpl.java | 2 --
.../src/main/java/org/apache/pulsar/broker/stats/DimensionStats.java | 2 --
.../pulsar/broker/stats/prometheus/metrics/LongAdderCounter.java | 2 --
.../java/org/apache/pulsar/compaction/CompactionRetentionTest.java | 4 +---
.../main/java/org/apache/pulsar/client/api/ClientConfiguration.java | 1 -
.../java/org/apache/pulsar/client/api/ConsumerConfiguration.java | 1 -
.../apache/pulsar/client/admin/internal/PulsarAdminBuilderImpl.java | 4
.../org/apache/pulsar/client/admin/internal/TransactionsImpl.java| 1 -
.../main/java/org/apache/pulsar/client/impl/AutoClusterFailover.java | 1 -
.../main/java/org/apache/pulsar/client/impl/ClientBuilderImpl.java | 2 --
.../src/main/java/org/apache/pulsar/client/impl/ConsumerBase.java| 4
.../main/java/org/apache/pulsar/client/impl/ConsumerBuilderImpl.java | 1 -
.../org/apache/pulsar/client/impl/ControlledClusterFailover.java | 1 -
.../main/java/org/apache/pulsar/client/impl/ReaderBuilderImpl.java | 1 -
.../org/apache/pulsar/client/impl/TransactionMetaStoreHandler.java | 2 --
.../java/org/apache/pulsar/client/impl/TypedMessageBuilderImpl.java | 1 -
.../pulsar/client/impl/transaction/TransactionBuilderImpl.java | 2 --
.../src/main/java/org/apache/pulsar/client/util/ObjectCache.java | 2 --
.../main/java/org/apache/bookkeeper/client/PulsarMockBookKeeper.java | 2 --
.../main/java/org/apache/bookkeeper/client/TestStatsProvider.java| 2 --
22 files changed, 2 insertions(+), 43 deletions(-)
diff --git
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedgerConfig.java
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedgerConfig.java
index 6ee9c2f9492..0c93a5b642c 100644
---
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedgerConfig.java
+++
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedgerConfig.java
@@ -170,7 +170,6 @@ public class ManagedLedgerConfig {
*the time unit
*/
public void setMinimumRolloverTime(int minimumRolloverTime, TimeUnit unit)
{
-checkArgument(minimumRolloverTime >= 0);
this.minimumRolloverTimeMs = (int) unit.toMillis(minimumRolloverTime);
checkArgument(maximumRolloverTimeMs >= minimumRolloverTimeMs,
"Minimum rollover time needs to be less than maximum rollover
time");
@@ -196,7 +195,6 @@ public class ManagedLedgerConfig {
*the time unit
*/
public void setMaximumRolloverTime(int maximumRolloverTime, TimeUnit unit)
{
-checkArgument(maximumRolloverTime >= 0);
this.maximumRolloverTimeMs = unit.toMillis(maximumRolloverTime);
checkArgument(maximumRolloverTimeMs >= minimumRolloverTimeMs,
"Maximum rollover time needs to be greater than minimum
rollover time");
@@ -413,8 +411,7 @@ public class ManagedLedgerConfig {
*time unit for retention time
*/
public ManagedLedgerConfig setRetentionTime(int retentionTime, TimeUnit
unit) {
-checkArgument(retentionTime >= -1, "The retention time should be -1, 0
or value > 0");
-this.retentionTimeMs = retentionTime != -1 ?
unit.toMillis(retentionTime) : -1;
+this.retentionTimeMs = unit.toMillis(retentionTime);
return this;
}
diff --git
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryMBeanImpl.java
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryMBeanImpl.java
index 5a6bc8017b7..cf3d7142d61 100644
---
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryMBeanImpl.java
+++
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryMBeanImpl.java
@@ -18,7 +18,6 @@
*/
package org.apache.bookkeeper.mledger.impl;
-import static com.google.common.base.Preconditions.checkArgument;
import java.util.concurrent.TimeUnit;
import
Re: [I] [Doc] Incomplete info re authenticationRefreshCheckSeconds [pulsar]
nodece commented on issue #22124: URL: https://github.com/apache/pulsar/issues/22124#issuecomment-1993884974 Thank you for sharing, you are right, the pulsar client stays connected, and I think this is a bug. Going to #22125. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch branch-3.1 updated: Bump to next snapshot version
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a commit to branch branch-3.1 in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/branch-3.1 by this push: new 8eb43cac5fb Bump to next snapshot version 8eb43cac5fb is described below commit 8eb43cac5fbc65117e39e160f91db654f950dc61 Author: Lari Hotari AuthorDate: Wed Mar 13 11:07:15 2024 +0200 Bump to next snapshot version --- bouncy-castle/bc/pom.xml | 2 +- bouncy-castle/bcfips-include-test/pom.xml | 2 +- bouncy-castle/bcfips/pom.xml | 2 +- bouncy-castle/pom.xml | 2 +- buildtools/pom.xml| 4 ++-- distribution/io/pom.xml | 2 +- distribution/offloaders/pom.xml | 2 +- distribution/pom.xml | 2 +- distribution/server/pom.xml | 2 +- distribution/shell/pom.xml| 2 +- docker/pom.xml| 2 +- docker/pulsar-all/pom.xml | 2 +- docker/pulsar/pom.xml | 2 +- jclouds-shaded/pom.xml| 2 +- managed-ledger/pom.xml| 2 +- pom.xml | 4 ++-- pulsar-broker-auth-athenz/pom.xml | 2 +- pulsar-broker-auth-oidc/pom.xml | 2 +- pulsar-broker-auth-sasl/pom.xml | 2 +- pulsar-broker-common/pom.xml | 2 +- pulsar-broker/pom.xml | 2 +- pulsar-client-1x-base/pom.xml | 2 +- pulsar-client-1x-base/pulsar-client-1x/pom.xml| 2 +- pulsar-client-1x-base/pulsar-client-2x-shaded/pom.xml | 2 +- pulsar-client-admin-api/pom.xml | 2 +- pulsar-client-admin-shaded/pom.xml| 2 +- pulsar-client-admin/pom.xml | 2 +- pulsar-client-all/pom.xml | 2 +- pulsar-client-api/pom.xml | 2 +- pulsar-client-auth-athenz/pom.xml | 2 +- pulsar-client-auth-sasl/pom.xml | 2 +- pulsar-client-messagecrypto-bc/pom.xml| 2 +- pulsar-client-shaded/pom.xml | 2 +- pulsar-client-tools-api/pom.xml | 2 +- pulsar-client-tools-customcommand-example/pom.xml | 2 +- pulsar-client-tools-test/pom.xml | 2 +- pulsar-client-tools/pom.xml | 2 +- pulsar-client/pom.xml | 2 +- pulsar-common/pom.xml | 2 +- pulsar-config-validation/pom.xml | 2 +- pulsar-docs-tools/pom.xml | 2 +- pulsar-functions/api-java/pom.xml | 2 +- pulsar-functions/instance/pom.xml | 2 +- pulsar-functions/java-examples-builtin/pom.xml| 2 +- pulsar-functions/java-examples/pom.xml| 2 +- pulsar-functions/localrun-shaded/pom.xml | 2 +- pulsar-functions/localrun/pom.xml | 2 +- pulsar-functions/pom.xml | 2 +- pulsar-functions/proto/pom.xml| 2 +- pulsar-functions/runtime-all/pom.xml | 2 +- pulsar-functions/runtime/pom.xml | 2 +- pulsar-functions/secrets/pom.xml | 2 +- pulsar-functions/utils/pom.xml| 2 +- pulsar-functions/worker/pom.xml | 2 +- pulsar-io/aerospike/pom.xml | 2 +- pulsar-io/alluxio/pom.xml | 2 +- pulsar-io/aws/pom.xml | 2 +- pulsar-io/batch-data-generator/pom.xml| 2 +- pulsar-io/batch-discovery-triggerers/pom.xml | 2 +- pulsar-io/canal/pom.xml | 2 +- pulsar-io/cassandra/pom.xml | 2 +- pulsar-io/common/pom.xml | 2 +- pulsar-io/core/pom.xml| 2 +- pulsar-io/data-generator/pom.xml | 2 +- pulsar-io/debezium/core/pom.xml | 2 +- pulsar-io/debezium/mongodb/pom.xml| 2 +- pulsar-io/debezium/mssql/pom.xml | 2 +- pulsar-io/debezium/mysql/pom.xml | 2 +- pulsar-io/debezium/oracle/pom.xml | 2 +- pulsar-io/debezium/pom.xml| 2 +- pulsar-io/debezium/postgres/pom.xml | 2 +- pulsar-io/docs/pom.xml| 2 +- pulsar-io/dynamodb/pom.xml
(pulsar) branch branch-3.2 updated: Bump to next snapshot version
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a commit to branch branch-3.2 in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/branch-3.2 by this push: new 30105c02604 Bump to next snapshot version 30105c02604 is described below commit 30105c02604c9bf3061ea445a0b3c3332b603801 Author: Lari Hotari AuthorDate: Wed Mar 13 11:08:01 2024 +0200 Bump to next snapshot version --- bouncy-castle/bc/pom.xml | 2 +- bouncy-castle/bcfips-include-test/pom.xml | 2 +- bouncy-castle/bcfips/pom.xml | 2 +- bouncy-castle/pom.xml | 2 +- buildtools/pom.xml| 4 ++-- distribution/io/pom.xml | 2 +- distribution/offloaders/pom.xml | 2 +- distribution/pom.xml | 2 +- distribution/server/pom.xml | 2 +- distribution/shell/pom.xml| 2 +- docker/pom.xml| 2 +- docker/pulsar-all/pom.xml | 2 +- docker/pulsar/pom.xml | 2 +- jclouds-shaded/pom.xml| 2 +- managed-ledger/pom.xml| 2 +- microbench/pom.xml| 2 +- pom.xml | 4 ++-- pulsar-bom/pom.xml| 2 +- pulsar-broker-auth-athenz/pom.xml | 2 +- pulsar-broker-auth-oidc/pom.xml | 2 +- pulsar-broker-auth-sasl/pom.xml | 2 +- pulsar-broker-common/pom.xml | 2 +- pulsar-broker/pom.xml | 2 +- pulsar-cli-utils/pom.xml | 2 +- pulsar-client-1x-base/pom.xml | 2 +- pulsar-client-1x-base/pulsar-client-1x/pom.xml| 2 +- pulsar-client-1x-base/pulsar-client-2x-shaded/pom.xml | 2 +- pulsar-client-admin-api/pom.xml | 2 +- pulsar-client-admin-shaded/pom.xml| 2 +- pulsar-client-admin/pom.xml | 2 +- pulsar-client-all/pom.xml | 2 +- pulsar-client-api/pom.xml | 2 +- pulsar-client-auth-athenz/pom.xml | 2 +- pulsar-client-auth-sasl/pom.xml | 2 +- pulsar-client-messagecrypto-bc/pom.xml| 2 +- pulsar-client-shaded/pom.xml | 2 +- pulsar-client-tools-api/pom.xml | 2 +- pulsar-client-tools-customcommand-example/pom.xml | 2 +- pulsar-client-tools-test/pom.xml | 2 +- pulsar-client-tools/pom.xml | 2 +- pulsar-client/pom.xml | 2 +- pulsar-common/pom.xml | 2 +- pulsar-config-validation/pom.xml | 2 +- pulsar-docs-tools/pom.xml | 2 +- pulsar-functions/api-java/pom.xml | 2 +- pulsar-functions/instance/pom.xml | 2 +- pulsar-functions/java-examples-builtin/pom.xml| 2 +- pulsar-functions/java-examples/pom.xml| 2 +- pulsar-functions/localrun-shaded/pom.xml | 2 +- pulsar-functions/localrun/pom.xml | 2 +- pulsar-functions/pom.xml | 2 +- pulsar-functions/proto/pom.xml| 2 +- pulsar-functions/runtime-all/pom.xml | 2 +- pulsar-functions/runtime/pom.xml | 2 +- pulsar-functions/secrets/pom.xml | 2 +- pulsar-functions/utils/pom.xml| 2 +- pulsar-functions/worker/pom.xml | 2 +- pulsar-io/aerospike/pom.xml | 2 +- pulsar-io/alluxio/pom.xml | 2 +- pulsar-io/aws/pom.xml | 2 +- pulsar-io/batch-data-generator/pom.xml| 2 +- pulsar-io/batch-discovery-triggerers/pom.xml | 2 +- pulsar-io/canal/pom.xml | 2 +- pulsar-io/cassandra/pom.xml | 2 +- pulsar-io/common/pom.xml | 2 +- pulsar-io/core/pom.xml| 2 +- pulsar-io/data-generator/pom.xml | 2 +- pulsar-io/debezium/core/pom.xml | 2 +- pulsar-io/debezium/mongodb/pom.xml| 2 +- pulsar-io/debezium/mssql/pom.xml | 2 +- pulsar-io/debezium/mysql/pom.xml | 2 +- pulsar-io/debezium/oracle/pom.xml | 2 +- pulsar-io/debezium/pom.xml
(pulsar) branch branch-2.11 updated: Bump to next snapshot version
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a commit to branch branch-2.11 in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/branch-2.11 by this push: new 5059bb3db45 Bump to next snapshot version 5059bb3db45 is described below commit 5059bb3db45c180f2e9a355dee900b91ba7628ea Author: Lari Hotari AuthorDate: Wed Mar 13 11:10:12 2024 +0200 Bump to next snapshot version --- bouncy-castle/bc/pom.xml | 2 +- bouncy-castle/bcfips-include-test/pom.xml | 2 +- bouncy-castle/bcfips/pom.xml | 2 +- bouncy-castle/pom.xml | 2 +- buildtools/pom.xml| 2 +- distribution/io/pom.xml | 2 +- distribution/offloaders/pom.xml | 2 +- distribution/pom.xml | 2 +- distribution/server/pom.xml | 2 +- distribution/shell/pom.xml| 2 +- docker/pom.xml| 2 +- docker/pulsar-all/pom.xml | 2 +- docker/pulsar/pom.xml | 2 +- jclouds-shaded/pom.xml| 2 +- kafka-connect-avro-converter-shaded/pom.xml | 2 +- managed-ledger/pom.xml| 2 +- pom.xml | 2 +- pulsar-broker-auth-athenz/pom.xml | 2 +- pulsar-broker-auth-sasl/pom.xml | 2 +- pulsar-broker-common/pom.xml | 2 +- pulsar-broker-shaded/pom.xml | 2 +- pulsar-broker/pom.xml | 2 +- pulsar-client-1x-base/pom.xml | 2 +- pulsar-client-1x-base/pulsar-client-1x/pom.xml| 2 +- pulsar-client-1x-base/pulsar-client-2x-shaded/pom.xml | 2 +- pulsar-client-admin-api/pom.xml | 2 +- pulsar-client-admin-shaded/pom.xml| 2 +- pulsar-client-admin/pom.xml | 2 +- pulsar-client-all/pom.xml | 2 +- pulsar-client-api/pom.xml | 2 +- pulsar-client-auth-athenz/pom.xml | 2 +- pulsar-client-auth-sasl/pom.xml | 2 +- pulsar-client-messagecrypto-bc/pom.xml| 2 +- pulsar-client-shaded/pom.xml | 2 +- pulsar-client-tools-api/pom.xml | 2 +- pulsar-client-tools-customcommand-example/pom.xml | 2 +- pulsar-client-tools-test/pom.xml | 2 +- pulsar-client-tools/pom.xml | 2 +- pulsar-client/pom.xml | 2 +- pulsar-common/pom.xml | 2 +- pulsar-config-validation/pom.xml | 2 +- pulsar-functions/api-java/pom.xml | 2 +- pulsar-functions/instance/pom.xml | 2 +- pulsar-functions/java-examples-builtin/pom.xml| 2 +- pulsar-functions/java-examples/pom.xml| 2 +- pulsar-functions/localrun-shaded/pom.xml | 2 +- pulsar-functions/localrun/pom.xml | 2 +- pulsar-functions/pom.xml | 2 +- pulsar-functions/proto/pom.xml| 2 +- pulsar-functions/runtime-all/pom.xml | 2 +- pulsar-functions/runtime/pom.xml | 2 +- pulsar-functions/secrets/pom.xml | 2 +- pulsar-functions/utils/pom.xml| 2 +- pulsar-functions/worker/pom.xml | 2 +- pulsar-io/aerospike/pom.xml | 2 +- pulsar-io/aws/pom.xml | 2 +- pulsar-io/batch-data-generator/pom.xml| 2 +- pulsar-io/batch-discovery-triggerers/pom.xml | 2 +- pulsar-io/canal/pom.xml | 2 +- pulsar-io/cassandra/pom.xml | 2 +- pulsar-io/common/pom.xml | 2 +- pulsar-io/core/pom.xml| 2 +- pulsar-io/data-generator/pom.xml | 2 +- pulsar-io/debezium/core/pom.xml | 2 +- pulsar-io/debezium/mongodb/pom.xml| 2 +- pulsar-io/debezium/mssql/pom.xml | 2 +- pulsar-io/debezium/mysql/pom.xml | 2 +- pulsar-io/debezium/oracle/pom.xml | 2 +- pulsar-io/debezium/pom.xml| 2 +- pulsar-io/debezium/postgres/pom.xml | 2 +- pulsar-io/docs/pom.xml| 2 +- pulsar-io/dynamodb/pom.xml| 2 +- pulsar-io/elastic-search/pom.xml
(pulsar) branch branch-3.0 updated: Bump to next snapshot version
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new 9cdc40442e3 Bump to next snapshot version 9cdc40442e3 is described below commit 9cdc40442e35c4537c704c08d5726caeb610fc1d Author: Lari Hotari AuthorDate: Wed Mar 13 11:06:28 2024 +0200 Bump to next snapshot version --- bouncy-castle/bc/pom.xml | 2 +- bouncy-castle/bcfips-include-test/pom.xml | 2 +- bouncy-castle/bcfips/pom.xml | 2 +- bouncy-castle/pom.xml | 2 +- buildtools/pom.xml| 4 ++-- distribution/io/pom.xml | 2 +- distribution/offloaders/pom.xml | 2 +- distribution/pom.xml | 2 +- distribution/server/pom.xml | 2 +- distribution/shell/pom.xml| 2 +- docker/pom.xml| 2 +- docker/pulsar-all/pom.xml | 2 +- docker/pulsar/pom.xml | 2 +- jclouds-shaded/pom.xml| 2 +- managed-ledger/pom.xml| 2 +- pom.xml | 4 ++-- pulsar-broker-auth-athenz/pom.xml | 2 +- pulsar-broker-auth-oidc/pom.xml | 2 +- pulsar-broker-auth-sasl/pom.xml | 2 +- pulsar-broker-common/pom.xml | 2 +- pulsar-broker/pom.xml | 2 +- pulsar-client-1x-base/pom.xml | 2 +- pulsar-client-1x-base/pulsar-client-1x/pom.xml| 2 +- pulsar-client-1x-base/pulsar-client-2x-shaded/pom.xml | 2 +- pulsar-client-admin-api/pom.xml | 2 +- pulsar-client-admin-shaded/pom.xml| 2 +- pulsar-client-admin/pom.xml | 2 +- pulsar-client-all/pom.xml | 2 +- pulsar-client-api/pom.xml | 2 +- pulsar-client-auth-athenz/pom.xml | 2 +- pulsar-client-auth-sasl/pom.xml | 2 +- pulsar-client-messagecrypto-bc/pom.xml| 2 +- pulsar-client-shaded/pom.xml | 2 +- pulsar-client-tools-api/pom.xml | 2 +- pulsar-client-tools-customcommand-example/pom.xml | 2 +- pulsar-client-tools-test/pom.xml | 2 +- pulsar-client-tools/pom.xml | 2 +- pulsar-client/pom.xml | 2 +- pulsar-common/pom.xml | 2 +- pulsar-config-validation/pom.xml | 2 +- pulsar-functions/api-java/pom.xml | 2 +- pulsar-functions/instance/pom.xml | 2 +- pulsar-functions/java-examples-builtin/pom.xml| 2 +- pulsar-functions/java-examples/pom.xml| 2 +- pulsar-functions/localrun-shaded/pom.xml | 2 +- pulsar-functions/localrun/pom.xml | 2 +- pulsar-functions/pom.xml | 2 +- pulsar-functions/proto/pom.xml| 2 +- pulsar-functions/runtime-all/pom.xml | 2 +- pulsar-functions/runtime/pom.xml | 2 +- pulsar-functions/secrets/pom.xml | 2 +- pulsar-functions/utils/pom.xml| 2 +- pulsar-functions/worker/pom.xml | 2 +- pulsar-io/aerospike/pom.xml | 2 +- pulsar-io/alluxio/pom.xml | 2 +- pulsar-io/aws/pom.xml | 2 +- pulsar-io/batch-data-generator/pom.xml| 2 +- pulsar-io/batch-discovery-triggerers/pom.xml | 2 +- pulsar-io/canal/pom.xml | 2 +- pulsar-io/cassandra/pom.xml | 2 +- pulsar-io/common/pom.xml | 2 +- pulsar-io/core/pom.xml| 2 +- pulsar-io/data-generator/pom.xml | 2 +- pulsar-io/debezium/core/pom.xml | 2 +- pulsar-io/debezium/mongodb/pom.xml| 2 +- pulsar-io/debezium/mssql/pom.xml | 2 +- pulsar-io/debezium/mysql/pom.xml | 2 +- pulsar-io/debezium/oracle/pom.xml | 2 +- pulsar-io/debezium/pom.xml| 2 +- pulsar-io/debezium/postgres/pom.xml | 2 +- pulsar-io/docs/pom.xml| 2 +- pulsar-io/dynamodb/pom.xml| 2 +- pulsar-io/elastic-search/pom.xml
Re: [I] [Bug] Not able to set default backlog-quota retention policy , default retention policy and default TTL [pulsar]
kunjbosamiaFalkonry commented on issue #6: URL: https://github.com/apache/pulsar/issues/6#issuecomment-1993698697 Nope it doesnt work for pulsar 3.0.3 as well (note : p here is a alias for /bin/pulsar-admin ) ``` pulsar@pulsar-toolset-0:/pulsar$ p namespaces create public/functions1 pulsar@pulsar-toolset-0:/pulsar$ p namespaces get-message-ttl public/functions1 null pulsar@pulsar-toolset-0:/pulsar$ p namespaces get-backlog-quotas public/functions1 pulsar@pulsar-toolset-0:/pulsar$ p namespaces get-retention public/functions1 null pulsar@pulsar-toolset-0:/pulsar$ ``` The default values I set in for broker in values.yaml under broker.configData-> ``` backlogQuotaDefaultRetentionPolicy: "consumer_backlog_eviction" backlogQuotaDefaultLimitBytes: "322122547200" backlogQuotaDefaultLimitSecond: "86400" defaultRetentionTimeInMinutes: "4320" defaultRetentionSizeInMB: "-1" ttlDurationDefaultInSeconds: "259200" topicLevelPoliciesEnabled: "true" ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch master updated: [improve] [broker] Servlet support response compression (#21667)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 7a4e16a8373 [improve] [broker] Servlet support response compression
(#21667)
7a4e16a8373 is described below
commit 7a4e16a8373754a6bc4713dcfe9d06c674ce3758
Author: Hang Chen
AuthorDate: Wed Mar 13 14:52:43 2024 +0800
[improve] [broker] Servlet support response compression (#21667)
---
.../org/apache/pulsar/broker/web/WebService.java | 16 +++--
.../apache/pulsar/broker/web/WebServiceTest.java | 72 ++
2 files changed, 82 insertions(+), 6 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebService.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebService.java
index 902593b7bf6..a7c42448990 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebService.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebService.java
@@ -44,6 +44,7 @@ import org.eclipse.jetty.server.handler.HandlerCollection;
import org.eclipse.jetty.server.handler.RequestLogHandler;
import org.eclipse.jetty.server.handler.ResourceHandler;
import org.eclipse.jetty.server.handler.StatisticsHandler;
+import org.eclipse.jetty.server.handler.gzip.GzipHandler;
import org.eclipse.jetty.servlet.FilterHolder;
import org.eclipse.jetty.servlet.ServletContextHandler;
import org.eclipse.jetty.servlet.ServletHolder;
@@ -258,15 +259,18 @@ public class WebService implements AutoCloseable {
public void addServlet(String path, ServletHolder servletHolder, boolean
requiresAuthentication,
Map attributeMap) {
-ServletContextHandler context = new
ServletContextHandler(ServletContextHandler.SESSIONS);
+ServletContextHandler servletContextHandler = new
ServletContextHandler(ServletContextHandler.SESSIONS);
// Notice: each context path should be unique, but there's nothing
here to verify that
-context.setContextPath(path);
-context.addServlet(servletHolder, MATCH_ALL);
+servletContextHandler.setContextPath(path);
+servletContextHandler.addServlet(servletHolder, MATCH_ALL);
if (attributeMap != null) {
-attributeMap.forEach(context::setAttribute);
+attributeMap.forEach(servletContextHandler::setAttribute);
}
-filterInitializer.addFilters(context, requiresAuthentication);
-handlers.add(context);
+filterInitializer.addFilters(servletContextHandler,
requiresAuthentication);
+
+GzipHandler gzipHandler = new GzipHandler();
+gzipHandler.setHandler(servletContextHandler);
+handlers.add(gzipHandler);
}
public void addStaticResources(String basePath, String resourcePath) {
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/web/WebServiceTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/web/WebServiceTest.java
index 8fb95eed789..72437fe3374 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/web/WebServiceTest.java
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/web/WebServiceTest.java
@@ -23,11 +23,14 @@ import static
org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsClient.
import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertTrue;
import static org.testng.Assert.fail;
+
import com.google.common.collect.Multimap;
import com.google.common.collect.Sets;
import com.google.common.io.CharStreams;
import com.google.common.io.Closeables;
import io.netty.handler.ssl.util.InsecureTrustManagerFactory;
+
+import java.io.BufferedReader;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
@@ -44,6 +47,8 @@ import java.util.HashSet;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.ZipException;
import javax.net.ssl.HttpsURLConnection;
import javax.net.ssl.KeyManager;
import javax.net.ssl.KeyManagerFactory;
@@ -354,6 +359,73 @@ public class WebServiceTest {
assertEquals(res.getResponseBody(), "ok");
}
+@Test
+public void testCompressOutputMetricsInPrometheus() throws Exception {
+
+setupEnv(true, false, false, false, -1, false);
+
+String metricsUrl = pulsar.getWebServiceAddress() + "/metrics/";
+
+String[] command = {"curl", "-H", "Accept-Encoding: gzip", metricsUrl};
+
+ProcessBuilder processBuilder = new ProcessBuilder(command);
+Process process = processBuilder.start();
+
+InputStream inputStream = process.getInputStream();
+
+try {
+GZIPInputStream gzipInputStream = new GZIPInputStream(inputStream);
+
+// Process the decompressed content
+StringBuilder content =
Re: [I] Add gzip compression before http service response [pulsar]
lhotari closed issue #16321: Add gzip compression before http service response URL: https://github.com/apache/pulsar/issues/16321 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [improve] [broker] Servlet support response compression [pulsar]
lhotari merged PR #21667: URL: https://github.com/apache/pulsar/pull/21667 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [improve][test] Reduce AdvertisedListenersMultiBrokerLeaderElectionTest flakiness [pulsar]
lhotari commented on PR #22258: URL: https://github.com/apache/pulsar/pull/22258#issuecomment-1993681693 > I'm not sure this fix could resolve the flakiness I'm not either. It is to reduce issues and this change isn't harmful. The thread leak detector reported leaked threads. The way this could help is that when the test fails, cleanup would happen properly and the test retry could pass with better chance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [cleanup][ml] ManagedCursor clean up. [pulsar]
codelipenghui commented on PR #22246: URL: https://github.com/apache/pulsar/pull/22246#issuecomment-1993629522 @dao-jun The PR touched the critical path of Pulsar, we should wait for more reviewers to review the PR. If a PR get merged with one reviewer and merged in a few hours, the committers will lost the chance to review your PR and new potential risks might introduce to the Pulsar codebase. Now, so many companies are relaying on the stable release, even if this is a trivial changes to cleanup the managed cursor, give more time for other reviewer will still benefit Pulsar. WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
