Re: [PR] [improve][broker] Taking message deduplication snapshot when closing topic [pulsar]
mattisonchao closed pull request #22498: [improve][broker] Taking message deduplication snapshot when closing topic URL: https://github.com/apache/pulsar/pull/22498 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [improve][broker] Taking message deduplication snapshot when closing topic [pulsar]
mattisonchao opened a new pull request, #22498: URL: https://github.com/apache/pulsar/pull/22498 ### Motivation The current implementation of message deduplication snapshot-taking may leave a very huge lag when traffic is high. if load balancer or someone tries to unload/load this topic the loading process may taking very long time. ### Modifications - Make message deduplication `snapshot` method return a future. - Support force taking snapshot without interval condition check. - Taking the snapshot when closing the topic ### Verifying this change - [x] Make sure that the change passes the CI checks. ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][broker] avoid offload system topic [pulsar]
mattisonchao closed pull request #22497: [fix][broker] avoid offload system topic URL: https://github.com/apache/pulsar/pull/22497 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [fix][broker] avoid offload system topic [pulsar]
mattisonchao opened a new pull request, #22497: URL: https://github.com/apache/pulsar/pull/22497 ### Motivation Avoid setting broker internal system topics using off-loader because some of them are the preconditions of other topics. The slow replying log speed will cause a delay in all the topic loading.(timeout) ### Modifications - Add a condition check for offload policies. ### Verifying this change - [x] Make sure that the change passes the CI checks. ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar-site) branch main updated: Docs sync done from apache/pulsar (#51ecd02)
This is an automated email from the ASF dual-hosted git repository. urfree pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/pulsar-site.git The following commit(s) were added to refs/heads/main by this push: new 7e4306ce33b2 Docs sync done from apache/pulsar (#51ecd02) 7e4306ce33b2 is described below commit 7e4306ce33b2c6a27ba5278e1ea909477c08adb0 Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> AuthorDate: Sat Apr 13 01:25:01 2024 + Docs sync done from apache/pulsar (#51ecd02) --- .../next/config/reference-configuration-broker.md | 17 + .../next/config/reference-configuration-standalone.md | 17 + 2 files changed, 34 insertions(+) diff --git a/static/reference/next/config/reference-configuration-broker.md b/static/reference/next/config/reference-configuration-broker.md index e0c8538a5f0d..b9017bd0df6c 100644 --- a/static/reference/next/config/reference-configuration-broker.md +++ b/static/reference/next/config/reference-configuration-broker.md @@ -3088,6 +3088,23 @@ Capacity for accept queue in the HTTP server Default is set to 8192. **Category**: Server +### httpServerGzipCompressionExcludedPaths +Gzip compression is enabled by default. Specific paths can be excluded from compression. +There are 2 syntaxes supported, Servlet url-pattern based, and Regex based. +If the spec starts with '^' the spec is assumed to be a regex based path spec and will match with normal Java regex rules. +If the spec starts with '/' then spec is assumed to be a Servlet url-pattern rules path spec for either an exact match or prefix based match. +If the spec starts with '*.' then spec is assumed to be a Servlet url-pattern rules path spec for a suffix based match. +All other syntaxes are unsupported. +Disable all compression with ^.* or ^.*$ + +**Type**: `java.util.List` + +**Default**: `[]` + +**Dynamic**: `false` + +**Category**: Server + ### httpServerThreadPoolQueueSize Capacity for thread pool queue in the HTTP server Default is set to 8192. diff --git a/static/reference/next/config/reference-configuration-standalone.md b/static/reference/next/config/reference-configuration-standalone.md index e0c8538a5f0d..b9017bd0df6c 100644 --- a/static/reference/next/config/reference-configuration-standalone.md +++ b/static/reference/next/config/reference-configuration-standalone.md @@ -3088,6 +3088,23 @@ Capacity for accept queue in the HTTP server Default is set to 8192. **Category**: Server +### httpServerGzipCompressionExcludedPaths +Gzip compression is enabled by default. Specific paths can be excluded from compression. +There are 2 syntaxes supported, Servlet url-pattern based, and Regex based. +If the spec starts with '^' the spec is assumed to be a regex based path spec and will match with normal Java regex rules. +If the spec starts with '/' then spec is assumed to be a Servlet url-pattern rules path spec for either an exact match or prefix based match. +If the spec starts with '*.' then spec is assumed to be a Servlet url-pattern rules path spec for a suffix based match. +All other syntaxes are unsupported. +Disable all compression with ^.* or ^.*$ + +**Type**: `java.util.List` + +**Default**: `[]` + +**Dynamic**: `false` + +**Category**: Server + ### httpServerThreadPoolQueueSize Capacity for thread pool queue in the HTTP server Default is set to 8192.
Re: [PR] [cleanup][broker] Remove unused NamespaceBundleFactory parameter when creating OwnershipCache [pulsar]
merlimat merged PR #22482: URL: https://github.com/apache/pulsar/pull/22482 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch master updated: [cleanup][broker] Remove unused NamespaceBundleFactory parameter when creating OwnershipCache (#22482)
This is an automated email from the ASF dual-hosted git repository.
mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 51ecd0235ce [cleanup][broker] Remove unused NamespaceBundleFactory
parameter when creating OwnershipCache (#22482)
51ecd0235ce is described below
commit 51ecd0235ce5d5ad03c563a3338b29c6a117d216
Author: Yunze Xu
AuthorDate: Sat Apr 13 08:58:53 2024 +0800
[cleanup][broker] Remove unused NamespaceBundleFactory parameter when
creating OwnershipCache (#22482)
---
.../pulsar/broker/namespace/NamespaceService.java | 2 +-
.../pulsar/broker/namespace/OwnershipCache.java | 4 +---
.../pulsar/broker/namespace/OwnershipCacheTest.java | 19 ---
3 files changed, 10 insertions(+), 15 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/NamespaceService.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/NamespaceService.java
index 4492f9c8094..7c62f264c78 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/NamespaceService.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/NamespaceService.java
@@ -192,7 +192,7 @@ public class NamespaceService implements AutoCloseable {
this.config = pulsar.getConfiguration();
this.loadManager = pulsar.getLoadManager();
this.bundleFactory = new NamespaceBundleFactory(pulsar,
Hashing.crc32());
-this.ownershipCache = new OwnershipCache(pulsar, bundleFactory, this);
+this.ownershipCache = new OwnershipCache(pulsar, this);
this.namespaceClients =
ConcurrentOpenHashMap.newBuilder().build();
this.bundleOwnershipListeners = new CopyOnWriteArrayList<>();
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/OwnershipCache.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/OwnershipCache.java
index 0033abf36c7..9a4534f5387 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/OwnershipCache.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/OwnershipCache.java
@@ -36,7 +36,6 @@ import java.util.concurrent.TimeoutException;
import lombok.extern.slf4j.Slf4j;
import org.apache.pulsar.broker.PulsarService;
import org.apache.pulsar.common.naming.NamespaceBundle;
-import org.apache.pulsar.common.naming.NamespaceBundleFactory;
import org.apache.pulsar.common.naming.NamespaceBundles;
import org.apache.pulsar.common.util.FutureUtil;
import org.apache.pulsar.metadata.api.coordination.LockManager;
@@ -115,8 +114,7 @@ public class OwnershipCache {
*
* the local broker URL that will be set as owner for the
ServiceUnit
*/
-public OwnershipCache(PulsarService pulsar, NamespaceBundleFactory
bundleFactory,
- NamespaceService namespaceService) {
+public OwnershipCache(PulsarService pulsar, NamespaceService
namespaceService) {
this.namespaceService = namespaceService;
this.pulsar = pulsar;
this.ownerBrokerUrl = pulsar.getBrokerServiceUrl();
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/namespace/OwnershipCacheTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/namespace/OwnershipCacheTest.java
index c92127457aa..2c3182659f0 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/namespace/OwnershipCacheTest.java
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/namespace/OwnershipCacheTest.java
@@ -55,15 +55,12 @@ import
org.apache.pulsar.metadata.api.extended.MetadataStoreExtended;
import org.apache.pulsar.metadata.coordination.impl.CoordinationServiceImpl;
import org.apache.pulsar.zookeeper.ZookeeperServerTest;
import org.awaitility.Awaitility;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;
@Test(groups = "broker")
public class OwnershipCacheTest {
-private static final Logger log =
LoggerFactory.getLogger(OwnershipCacheTest.class);
private PulsarService pulsar;
private ServiceConfiguration config;
@@ -123,14 +120,14 @@ public class OwnershipCacheTest {
@Test
public void testConstructor() {
-OwnershipCache cache = new OwnershipCache(this.pulsar, bundleFactory,
nsService);
+OwnershipCache cache = new OwnershipCache(this.pulsar, nsService);
assertNotNull(cache);
assertNotNull(cache.getOwnedBundles());
}
@Test
public void testDisableOwnership() throws Exception {
-OwnershipCache cache = new OwnershipCache(this.pulsar, bundleFactory,
nsService);
+OwnershipCache cache = new OwnershipCache(this.pulsar, nsService);
NamespaceBundle testBundle = new
Re: [PR] Added a split example [pulsar-site]
heesung-sn commented on code in PR #884: URL: https://github.com/apache/pulsar-site/pull/884#discussion_r1563409436 ## docs/concepts-broker-load-balancing-concepts.md: ## @@ -173,7 +173,14 @@ Based on the broker resource usage (for example, the number of topics or session Step 3: split bundles at the specific boundaries from step 2. -For how to split bundles manually, see TBD (the docs is WIP, stay tuned!). +For how to split bundles manually, please refer to [split-bundle](https://pulsar.apache.org/reference/#/3.2.x/pulsar-admin/namespaces?id=split-bundle) admin command. + Review Comment: thanks. updated -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar-site) branch heesung-sn-patch-1 updated (56a9d0aa7a87 -> 5197655389cb)
This is an automated email from the ASF dual-hosted git repository. heesung pushed a change to branch heesung-sn-patch-1 in repository https://gitbox.apache.org/repos/asf/pulsar-site.git from 56a9d0aa7a87 Update concepts-broker-load-balancing-concepts.md add 5197655389cb Update concepts-broker-load-balancing-concepts.md No new revisions were added by this update. Summary of changes: docs/concepts-broker-load-balancing-concepts.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
(pulsar-site) branch heesung-sn-patch-1 updated (c15dd4bb69f0 -> 56a9d0aa7a87)
This is an automated email from the ASF dual-hosted git repository. heesung pushed a change to branch heesung-sn-patch-1 in repository https://gitbox.apache.org/repos/asf/pulsar-site.git from c15dd4bb69f0 Update concepts-broker-load-balancing-concepts.md add 56a9d0aa7a87 Update concepts-broker-load-balancing-concepts.md No new revisions were added by this update. Summary of changes: docs/concepts-broker-load-balancing-concepts.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Re: [PR] Added a split example [pulsar-site]
dragosvictor commented on code in PR #884: URL: https://github.com/apache/pulsar-site/pull/884#discussion_r1563341391 ## docs/concepts-broker-load-balancing-concepts.md: ## @@ -173,7 +173,14 @@ Based on the broker resource usage (for example, the number of topics or session Step 3: split bundles at the specific boundaries from step 2. -For how to split bundles manually, see TBD (the docs is WIP, stay tuned!). +For how to split bundles manually, please refer to [split-bundle](https://pulsar.apache.org/reference/#/3.2.x/pulsar-admin/namespaces?id=split-bundle) admin command. + Review Comment: This link would point to a hardcoded versioned URL, where we should use the URL relative to each document version. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar-site) branch heesung-sn-patch-1 updated (cab129ac2a9f -> c15dd4bb69f0)
This is an automated email from the ASF dual-hosted git repository. heesung pushed a change to branch heesung-sn-patch-1 in repository https://gitbox.apache.org/repos/asf/pulsar-site.git from cab129ac2a9f Added a split example add c15dd4bb69f0 Update concepts-broker-load-balancing-concepts.md No new revisions were added by this update. Summary of changes: docs/concepts-broker-load-balancing-concepts.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
[PR] Added a split example [pulsar-site]
heesung-sn opened a new pull request, #884: URL: https://github.com/apache/pulsar-site/pull/884 ### ✅ Contribution Checklist - [ ] I read the [contribution guide](https://pulsar.apache.org/contribute/document-contribution/) - [ ] I updated the [versioned docs](https://pulsar.apache.org/contribute/document-contribution/#update-versioned-docs) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar-site) branch heesung-sn-patch-1 created (now cab129ac2a9f)
This is an automated email from the ASF dual-hosted git repository. heesung pushed a change to branch heesung-sn-patch-1 in repository https://gitbox.apache.org/repos/asf/pulsar-site.git at cab129ac2a9f Added a split example This branch includes the following new commits: new cab129ac2a9f Added a split example The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(pulsar-site) 01/01: Added a split example
This is an automated email from the ASF dual-hosted git repository. heesung pushed a commit to branch heesung-sn-patch-1 in repository https://gitbox.apache.org/repos/asf/pulsar-site.git commit cab129ac2a9f8598c59369869bacb2c1dab0d597 Author: Heesung Sohn <103456639+heesung...@users.noreply.github.com> AuthorDate: Fri Apr 12 16:11:50 2024 -0700 Added a split example --- docs/concepts-broker-load-balancing-concepts.md | 8 +++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/concepts-broker-load-balancing-concepts.md b/docs/concepts-broker-load-balancing-concepts.md index 2b7d5a1f1f1c..9ee873deb850 100644 --- a/docs/concepts-broker-load-balancing-concepts.md +++ b/docs/concepts-broker-load-balancing-concepts.md @@ -173,7 +173,13 @@ Based on the broker resource usage (for example, the number of topics or session Step 3: split bundles at the specific boundaries from step 2. -For how to split bundles manually, see TBD (the docs is WIP, stay tuned!). +For how to split bundles manually, please refer to [split-bundle](https://pulsar.apache.org/reference/#/next/pulsar-admin/namespaces?id=split-bundle) admin command. + +Example: + +`pulsar-admin namespaces split-bundle -b LARGEST -san topic_count_equally_divide -u tenant/largest_namespace` +* Specify a specific bundle at -b, if you already know the target bundle to split. +
Re: [PR] [fix][broker] Fix Replicated Topic unload bug when ExtensibleLoadManager is enabled [pulsar]
github-actions[bot] commented on PR #22496: URL: https://github.com/apache/pulsar/pull/22496#issuecomment-2052652209 @heesung-sn Please add the following content to your PR description and select a checkbox: ``` - [ ] `doc` - [ ] `doc-required` - [ ] `doc-not-needed` - [ ] `doc-complete` ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [fix][broker] Fix Replicated Topic unload bug when ExtensibleLoadManager is enabled [pulsar]
heesung-sn opened a new pull request, #22496: URL: https://github.com/apache/pulsar/pull/22496 ### Motivation Noticed that Replicated Topic is not unloaded upon setNamespaceReplicationClusters when ExtensibleLoadManager is enabled. ### Modifications - Fix unloading bundles upon namespace policy update. (isNamespaceBundleOwned didn't check the bundle ownership properly when ExtensibleLoadManager is enabled.) - Do not replicate ExtensibleLoadManager internal topics (bundle state, broker and bundle load data topics) because they are only used for the local cluster. - Copy ReplicatorGlobalNSTest for ExtensibleLoadManager. ### Verifying this change - [x] Make sure that the change passes the CI checks. ### Does this pull request potentially affect one of the following parts: *If the box was checked, please highlight the changes* - [ ] Dependencies (add or upgrade a dependency) - [ ] The public API - [ ] The schema - [ ] The default values of configurations - [ ] The threading model - [ ] The binary protocol - [ ] The REST endpoints - [ ] The admin CLI options - [ ] The metrics - [ ] Anything that affects deployment ### Documentation - [ ] `doc` - [ ] `doc-required` - [ ] `doc-not-needed` - [ ] `doc-complete` ### Matching PR in forked repository PR in forked repository: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][broker] Optimize /metrics, fix unbounded request queue issue and fix race conditions in metricsBufferResponse mode [pulsar]
lhotari commented on PR #22494: URL: https://github.com/apache/pulsar/pull/22494#issuecomment-2052438415 I did allocation profiling with Async Profiler. Used similar commands as in https://github.com/apache/pulsar/pull/22494#issuecomment-2052043146 , but with async profiler activated with `OPTS`. ```shell OPTS="-agentpath:$HOME/tools/async-profiler/lib/libasyncProfiler.so=start,event=cpu,alloc=2m,lock=10ms,file=$PWD/profile.jfr" PULSAR_MEM="-Xms2g -Xmx4g -XX:MaxDirectMemorySize=6g" PULSAR_GC="-XX:+UseG1GC -XX:+PerfDisableSharedMem -XX:+AlwaysPreTouch" PULSAR_EXTRA_OPTS="-Djute.maxbuffer=2000" PULSAR_STANDALONE_USE_ZOOKEEPER=1 bin/pulsar standalone -nss -nfw 2>&1 | tee standalone.log ``` With `OPTS="-agentpath:$HOME/tools/async-profiler/lib/libasyncProfiler.so=start,event=cpu,alloc=2m,lock=10ms,file=$PWD/profile.jfr"`, it's possible to do CPU, allocation and lock profiling all at once. I run async-profiler on a Linux box to get best accuracy. After profiling, I then use this shell script function to generate multiple flamegraph htmls out of the JFR file: https://github.com/lhotari/pulsar-contributor-toolbox/blob/c150c3d9afc23d4865c2e3283c087e1c1261b4ee/functions/pulsar-contributor-toolbox-functions.sh#L1438-L1458 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][broker] Optimize /metrics, fix unbounded request queue issue and fix race conditions in metricsBufferResponse mode [pulsar]
lhotari commented on PR #22494:
URL: https://github.com/apache/pulsar/pull/22494#issuecomment-2052417670
Most recent test run without turning on the caching:
```
❯ k6 run load_test.js
/\ |‾‾| /‾‾/ /‾‾/
/\ / \ | |/ / / /
/ \/\| ( / ‾‾\
/ \ | |\ \ | (‾) |
/ __ \ |__| \__\ \_/ .io
execution: local
script: load_test.js
output: -
scenarios: (100.00%) 1 scenario, 100 max VUs, 10m30s max duration
(incl. graceful stop):
* default: 1 iterations shared among 100 VUs (maxDuration:
10m0s, gracefulStop: 30s)
data_received..: 3.8 TB 8.5 GB/s
data_sent..: 880 kB 2.0 kB/s
http_req_blocked...: avg=42.58µs min=1µs med=3µs
max=4.93ms p(90)=5µs p(95)=7µs
http_req_connecting: avg=33.7µs min=0s med=0s
max=4ms p(90)=0s p(95)=0s
http_req_duration..: avg=4.46smin=298.14ms med=4.44s
max=8.7s p(90)=7.8s p(95)=8.2s
{ expected_response:true }...: avg=4.46smin=298.14ms med=4.44s
max=8.7s p(90)=7.8s p(95)=8.2s
http_req_failed: 0.00% ✓ 0 ✗ 1
http_req_receiving.: avg=178.21ms min=75.75ms med=176.97ms
max=359.87ms p(90)=204.48ms p(95)=213.44ms
http_req_sending...: avg=37.42µs min=4µs med=13µs
max=25.6ms p(90)=25µs p(95)=33µs
http_req_tls_handshaking...: avg=0s min=0s med=0s
max=0s p(90)=0s p(95)=0s
http_req_waiting...: avg=4.29smin=143.93ms med=4.26s
max=8.56sp(90)=7.62sp(95)=8.03s
http_reqs..: 1 22.281335/s
iteration_duration.: avg=4.46smin=298.19ms med=4.44s
max=8.7s p(90)=7.8s p(95)=8.2s
iterations.: 1 22.281335/s
vus: 22 min=22 max=100
vus_max: 100min=100 max=100
running (07m28.8s), 000/100 VUs, 1 complete and 0 interrupted iterations
default ✓ [==] 100 VUs 07m28.8s/10m0s
1/1 shared iters
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [fix][broker] Optimize /metrics, fix unbounded request queue issue and fix race conditions in metricsBufferResponse mode [pulsar]
lhotari commented on code in PR #22494:
URL: https://github.com/apache/pulsar/pull/22494#discussion_r1563054260
##
pulsar-common/src/main/java/org/apache/pulsar/common/util/SimpleTextOutputStream.java:
##
@@ -44,7 +46,8 @@ public SimpleTextOutputStream write(byte[] a, int offset, int
len) {
}
public SimpleTextOutputStream write(char c) {
-write(String.valueOf(c));
+singleCharBuffer.put(0, c);
+buffer.writeCharSequence(singleCharBuffer, CharsetUtil.UTF_8);
Review Comment:
@merlimat optimization added in bf1e994
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [fix][broker] Optimize /metrics, fix unbounded request queue issue and fix race conditions in metricsBufferResponse mode [pulsar]
lhotari commented on code in PR #22494:
URL: https://github.com/apache/pulsar/pull/22494#discussion_r1563046536
##
pulsar-common/src/main/java/org/apache/pulsar/common/util/SimpleTextOutputStream.java:
##
@@ -44,7 +46,8 @@ public SimpleTextOutputStream write(byte[] a, int offset, int
len) {
}
public SimpleTextOutputStream write(char c) {
-write(String.valueOf(c));
+singleCharBuffer.put(0, c);
+buffer.writeCharSequence(singleCharBuffer, CharsetUtil.UTF_8);
Review Comment:
@merlimat good point. I'll add the writeByte optimization for the ascii
character range where it's safe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
(pulsar) branch master updated: [fix][test] Move ExtensibleLoadManagerImplTest to flaky tests (#22495)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 97153dcf6f0 [fix][test] Move ExtensibleLoadManagerImplTest to flaky
tests (#22495)
97153dcf6f0 is described below
commit 97153dcf6f0f958a2ee2816ea64115a9a9ef4635
Author: Lari Hotari
AuthorDate: Fri Apr 12 11:38:42 2024 -0700
[fix][test] Move ExtensibleLoadManagerImplTest to flaky tests (#22495)
---
.../broker/loadbalance/extensions/ExtensibleLoadManagerImplTest.java| 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImplTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImplTest.java
index aee57f9d260..e87d6c994cd 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImplTest.java
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImplTest.java
@@ -142,7 +142,7 @@ import org.testng.annotations.Test;
* Unit test for {@link ExtensibleLoadManagerImpl}.
*/
@Slf4j
-@Test(groups = "broker")
+@Test(groups = "flaky")
@SuppressWarnings("unchecked")
public class ExtensibleLoadManagerImplTest extends
ExtensibleLoadManagerImplBaseTest {
Re: [PR] [fix][test] Move ExtensibleLoadManagerImplTest to flaky tests [pulsar]
lhotari merged PR #22495: URL: https://github.com/apache/pulsar/pull/22495 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][broker] Optimize /metrics, fix unbounded request queue issue and fix race conditions in metricsBufferResponse mode [pulsar]
merlimat commented on code in PR #22494:
URL: https://github.com/apache/pulsar/pull/22494#discussion_r1562981115
##
pulsar-common/src/main/java/org/apache/pulsar/common/util/SimpleTextOutputStream.java:
##
@@ -44,7 +46,8 @@ public SimpleTextOutputStream write(byte[] a, int offset, int
len) {
}
public SimpleTextOutputStream write(char c) {
-write(String.valueOf(c));
+singleCharBuffer.put(0, c);
+buffer.writeCharSequence(singleCharBuffer, CharsetUtil.UTF_8);
Review Comment:
I think the original `writeByte()` was appropriate here for the single
char. This is only used for `{`, `'` and `"` characters, it will never have any
non ascii chars
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
(pulsar) branch master updated: [improve][broker] Improve Gzip compression, allow excluding specific paths or disabling it (#22370)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 15ed6595af5 [improve][broker] Improve Gzip compression, allow
excluding specific paths or disabling it (#22370)
15ed6595af5 is described below
commit 15ed6595af5489a007db82002ed3391589bad54d
Author: Lari Hotari
AuthorDate: Fri Apr 12 10:35:09 2024 -0700
[improve][broker] Improve Gzip compression, allow excluding specific paths
or disabling it (#22370)
---
.../apache/pulsar/broker/ServiceConfiguration.java | 13 ++
.../apache/pulsar/broker/web/GzipHandlerUtil.java | 48 ++
.../org/apache/pulsar/broker/web/WebService.java | 10 ++---
.../pulsar/proxy/server/AdminProxyHandler.java | 1 +
4 files changed, 66 insertions(+), 6 deletions(-)
diff --git
a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
index a7deda752fd..38a4c552f0b 100644
---
a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
+++
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
@@ -331,6 +331,19 @@ public class ServiceConfiguration implements
PulsarConfiguration {
+ "(0 to disable limiting)")
private int maxHttpServerConnections = 2048;
+@FieldContext(category = CATEGORY_SERVER, doc =
+"Gzip compression is enabled by default. Specific paths can be
excluded from compression.\n"
++ "There are 2 syntaxes supported, Servlet url-pattern
based, and Regex based.\n"
++ "If the spec starts with '^' the spec is assumed to be a
regex based path spec and will match "
++ "with normal Java regex rules.\n"
++ "If the spec starts with '/' then spec is assumed to be
a Servlet url-pattern rules path spec "
++ "for either an exact match or prefix based match.\n"
++ "If the spec starts with '*.' then spec is assumed to be
a Servlet url-pattern rules path spec "
++ "for a suffix based match.\n"
++ "All other syntaxes are unsupported.\n"
++ "Disable all compression with ^.* or ^.*$")
+private List httpServerGzipCompressionExcludedPaths = new
ArrayList<>();
+
@FieldContext(category = CATEGORY_SERVER, doc = "Whether to enable the
delayed delivery for messages.")
private boolean delayedDeliveryEnabled = true;
diff --git
a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/web/GzipHandlerUtil.java
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/web/GzipHandlerUtil.java
new file mode 100644
index 000..37c9c05e5d5
--- /dev/null
+++
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/web/GzipHandlerUtil.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.web;
+
+import java.util.List;
+import org.eclipse.jetty.server.Handler;
+import org.eclipse.jetty.server.handler.gzip.GzipHandler;
+
+public class GzipHandlerUtil {
+public static Handler wrapWithGzipHandler(Handler innerHandler,
List gzipCompressionExcludedPaths) {
+Handler wrappedHandler;
+if (isGzipCompressionCompletelyDisabled(gzipCompressionExcludedPaths))
{
+// no need to add GZIP handler if it's disabled by setting the
excluded path to "^.*" or "^.*$"
+wrappedHandler = innerHandler;
+} else {
+// add GZIP handler which is active when the request contains
"Accept-Encoding: gzip" header
+GzipHandler gzipHandler = new GzipHandler();
+gzipHandler.setHandler(innerHandler);
+if (gzipCompressionExcludedPaths != null &&
gzipCompressionExcludedPaths.size() > 0) {
+
gzipHandler.setExcludedPaths(gzipCompressionExcludedPaths.toArray(new
String[0]));
+}
+wrappedHandler = gzipHandler;
+}
+
Re: [PR] [improve][broker] Improve Gzip compression, allow excluding specific paths or disabling it [pulsar]
lhotari merged PR #22370: URL: https://github.com/apache/pulsar/pull/22370 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][broker] Optimize /metrics, fix unbounded request queue issue and fix race conditions in metricsBufferResponse mode [pulsar]
lhotari commented on code in PR #22494:
URL: https://github.com/apache/pulsar/pull/22494#discussion_r1562901480
##
pulsar-common/src/main/java/org/apache/pulsar/common/util/SimpleTextOutputStream.java:
##
@@ -44,7 +46,8 @@ public SimpleTextOutputStream write(byte[] a, int offset, int
len) {
}
public SimpleTextOutputStream write(char c) {
-write(String.valueOf(c));
+singleCharBuffer.put(0, c);
+buffer.writeCharSequence(singleCharBuffer, CharsetUtil.UTF_8);
Review Comment:
This change reduces a lot of String allocations. This was visible in
allocation profiling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
(pulsar) branch master updated: [fix][test] Fix NPE in BookKeeperClusterTestCase tearDown (#22493)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new d1748573f1c [fix][test] Fix NPE in BookKeeperClusterTestCase tearDown
(#22493)
d1748573f1c is described below
commit d1748573f1cb294838b69b5d80af672c3ee9e453
Author: Lari Hotari
AuthorDate: Fri Apr 12 10:09:54 2024 -0700
[fix][test] Fix NPE in BookKeeperClusterTestCase tearDown (#22493)
---
.../org/apache/bookkeeper/test/BookKeeperClusterTestCase.java | 8 +---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git
a/managed-ledger/src/test/java/org/apache/bookkeeper/test/BookKeeperClusterTestCase.java
b/managed-ledger/src/test/java/org/apache/bookkeeper/test/BookKeeperClusterTestCase.java
index 0ddd04ebc48..a323ecfeb8e 100644
---
a/managed-ledger/src/test/java/org/apache/bookkeeper/test/BookKeeperClusterTestCase.java
+++
b/managed-ledger/src/test/java/org/apache/bookkeeper/test/BookKeeperClusterTestCase.java
@@ -86,7 +86,7 @@ public abstract class BookKeeperClusterTestCase {
protected String testName;
-@BeforeMethod
+@BeforeMethod(alwaysRun = true)
public void handleTestMethodName(Method method) {
testName = method.getName();
}
@@ -148,7 +148,7 @@ public abstract class BookKeeperClusterTestCase {
}
}
-@BeforeTest
+@BeforeTest(alwaysRun = true)
public void setUp() throws Exception {
setUp(getLedgersRootPath());
}
@@ -222,7 +222,9 @@ public abstract class BookKeeperClusterTestCase {
tearDownException = e;
}
-executor.shutdownNow();
+if (executor != null) {
+executor.shutdownNow();
+}
LOG.info("Tearing down test {} in {} ms.", testName,
sw.elapsed(TimeUnit.MILLISECONDS));
if (tearDownException != null) {
Re: [PR] [fix][test] Fix NPE in BookKeeperClusterTestCase tearDown [pulsar]
lhotari merged PR #22493: URL: https://github.com/apache/pulsar/pull/22493 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [fix][test] Move ExtensibleLoadManagerImplTest to flaky tests [pulsar]
lhotari opened a new pull request, #22495: URL: https://github.com/apache/pulsar/pull/22495 ### Motivation - this test is flaky, let's categorize it as flaky until it is stable - logs for [most recent case](https://github.com/apache/pulsar/actions/runs/8664728390/job/23762246664?pr=22494#step:11:716): https://gist.github.com/lhotari/25d618984841eae4b7fd51ae3908e5d7 ### Modifications - Move ExtensibleLoadManagerImplTest to flaky tests ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch master updated: [improve][broker] backlog quota exceed limit log replaced with `debug` (#22488)
This is an automated email from the ASF dual-hosted git repository.
nicoloboschi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new b85730069ee [improve][broker] backlog quota exceed limit log replaced
with `debug` (#22488)
b85730069ee is described below
commit b85730069ee4c5f96406a075e354d0592fdab434
Author: Mukesh Kumar <65598381+mukesh...@users.noreply.github.com>
AuthorDate: Fri Apr 12 22:07:28 2024 +0530
[improve][broker] backlog quota exceed limit log replaced with `debug`
(#22488)
---
.../org/apache/pulsar/broker/service/persistent/PersistentTopic.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
index 3ceecd7f4aa..3c9ab04d79a 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
@@ -3224,14 +3224,14 @@ public class PersistentTopic extends AbstractTopic
implements Topic, AddEntryCal
if ((retentionPolicy ==
BacklogQuota.RetentionPolicy.producer_request_hold
|| retentionPolicy ==
BacklogQuota.RetentionPolicy.producer_exception)) {
if (backlogQuotaType == BacklogQuotaType.destination_storage
&& isSizeBacklogExceeded()) {
-log.info("[{}] Size backlog quota exceeded. Cannot create
producer [{}]", this.getName(),
+log.debug("[{}] Size backlog quota exceeded. Cannot create
producer [{}]", this.getName(),
producerName);
return FutureUtil.failedFuture(new
TopicBacklogQuotaExceededException(retentionPolicy));
}
if (backlogQuotaType == BacklogQuotaType.message_age) {
return checkTimeBacklogExceeded().thenCompose(isExceeded
-> {
if (isExceeded) {
-log.info("[{}] Time backlog quota exceeded. Cannot
create producer [{}]", this.getName(),
+log.debug("[{}] Time backlog quota exceeded.
Cannot create producer [{}]", this.getName(),
producerName);
return FutureUtil.failedFuture(new
TopicBacklogQuotaExceededException(retentionPolicy));
} else {
Re: [PR] [improve][broker] backlog quota exceed limit log replaced with `debug` [pulsar]
nicoloboschi merged PR #22488: URL: https://github.com/apache/pulsar/pull/22488 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][broker] Optimize /metrics, fix unbounded request queue issue and fix race conditions in metricsBufferResponse mode [pulsar]
lhotari commented on PR #22494: URL: https://github.com/apache/pulsar/pull/22494#issuecomment-2052045246 One interesting detail is that in the load test, the system couldn't keep up with the load without making the optimization to direct buffer allocation in commit eb53342. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][broker] Optimize /metrics, fix unbounded request queue issue and fix race conditions in metricsBufferResponse mode [pulsar]
lhotari commented on PR #22494:
URL: https://github.com/apache/pulsar/pull/22494#issuecomment-2052043146
Performance test results with the changes.
Reproducing the results:
```shell
gh pr checkout 22494
mvn -Pcore-modules,-main -T 1C clean install -DskipTests -Dspotbugs.skip=true
rm -rf data
PULSAR_MEM="-Xms2g -Xmx4g -XX:MaxDirectMemorySize=6g"
PULSAR_GC="-XX:+UseG1GC -XX:+PerfDisableSharedMem -XX:+AlwaysPreTouch"
PULSAR_EXTRA_OPTS="-Djute.maxbuffer=2000" PULSAR_STANDALONE_USE_ZOOKEEPER=1
bin/pulsar standalone -nss -nfw 2>&1 | tee standalone.log
```
```shell
git clone https://github.com/lhotari/pulsar-playground
cd pulsar-playground
./gradlew shadowJar
# create the topics
java -cp build/libs/pulsar-playground-all.jar
com.github.lhotari.pulsar.playground.TestScenarioCreateLongNamedTopics
```
```
git clone https://github.com/lhotari/pulsar-playground
cd pulsar-playground/experiments/metrics-performance
❯ k6 run load_test.js
/\ |‾‾| /‾‾/ /‾‾/
/\ / \ | |/ / / /
/ \/\| ( / ‾‾\
/ \ | |\ \ | (‾) |
/ __ \ |__| \__\ \_/ .io
execution: local
script: load_test.js
output: -
scenarios: (100.00%) 1 scenario, 100 max VUs, 10m30s max duration
(incl. graceful stop):
* default: 1 iterations shared among 100 VUs (maxDuration:
10m0s, gracefulStop: 30s)
data_received..: 3.8 TB 8.1 GB/s
data_sent..: 880 kB 1.9 kB/s
http_req_blocked...: avg=35.04µs min=1µs med=3µs
max=3.92ms p(90)=6µs p(95)=8µs
http_req_connecting: avg=25.17µs min=0s med=0s
max=3.02ms p(90)=0s p(95)=0s
http_req_duration..: avg=4.7s min=457.09ms med=4.68s
max=9.71sp(90)=8.15sp(95)=8.58s
{ expected_response:true }...: avg=4.7s min=457.09ms med=4.68s
max=9.71sp(90)=8.15sp(95)=8.58s
http_req_failed: 0.00% ✓ 0 ✗ 1
http_req_receiving.: avg=185.35ms min=86.9ms med=183.9ms
max=469.76ms p(90)=214.08ms p(95)=224.08ms
http_req_sending...: avg=58.44µs min=3µs med=15µs
max=41.26ms p(90)=28µs p(95)=38µs
http_req_tls_handshaking...: avg=0s min=0s med=0s
max=0s p(90)=0s p(95)=0s
http_req_waiting...: avg=4.51smin=302.02ms med=4.5s
max=9.56sp(90)=7.96sp(95)=8.4s
http_reqs..: 1 21.179615/s
iteration_duration.: avg=4.7s min=457.16ms med=4.69s
max=9.71sp(90)=8.15sp(95)=8.58s
iterations.: 1 21.179615/s
vus: 5 min=5 max=100
vus_max: 100min=100 max=100
running (07m52.2s), 000/100 VUs, 1 complete and 0 interrupted iterations
default ✓ [==] 100 VUs 07m52.2s/10m0s
1/1 shared iters
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [fix][broker] Support lookup options for extensible load manager [pulsar]
heesung-sn commented on code in PR #22487:
URL: https://github.com/apache/pulsar/pull/22487#discussion_r1562705484
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImpl.java:
##
@@ -544,22 +550,19 @@ private CompletableFuture>
getBrokerLookupData(
String bundle) {
return owner.thenCompose(broker -> {
if (broker.isEmpty()) {
-String errorMsg = String.format(
-"Failed to get or assign the owner for bundle:%s",
bundle);
-log.error(errorMsg);
-throw new IllegalStateException(errorMsg);
-}
-return CompletableFuture.completedFuture(broker.get());
-}).thenCompose(broker ->
this.getBrokerRegistry().lookupAsync(broker).thenCompose(brokerLookupData -> {
-if (brokerLookupData.isEmpty()) {
-String errorMsg = String.format(
-"Failed to lookup broker:%s for bundle:%s, the broker
has not been registered.",
-broker, bundle);
-log.error(errorMsg);
-throw new IllegalStateException(errorMsg);
+return CompletableFuture.completedFuture(Optional.empty());
Review Comment:
nit: Shouldn't we return the IllegalStateException?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[PR] [fix][broker] Optimize /metrics, fix unbounded request queue issue and fix race conditions in metricsBufferResponse mode [pulsar]
lhotari opened a new pull request, #22494: URL: https://github.com/apache/pulsar/pull/22494 Fixes #22477 ### Motivation There are multiple problems in the /metrics endpoint: - requests are handled one-by-one and added into a queue - when requests have timed out, they get processed regardless. time outs are detected in the later phases of the processing. Instead, timed out requests should be short-circuited in the beginning of the processing. - processing is single threaded and therefore the throughput is low - the metricsBufferResponse mode added in #14453 improves things, but it contains some race conditions where a buffer can get released when it is in use. ### Modifications - fix the request timeout issue - enable multithreading. currently fixed to 4 threads since higher concurrency requires more memory and leads to OOM issues. - by default, combine concurrent requests to reuse the same result. this improves throughput even when `metricsBufferResponse` mode isn't enabled - remove the previous `metricsBufferResponse` solution since the TimeWindow and WindowWrap classes arean't needed at all. The concurrent request combining solution can also be used for `metricsBufferResponse` solution. - optimize some details of stats generation which were allocating a lot of objects. ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][test] Fix NPE in BookKeeperClusterTestCase tearDown [pulsar]
lhotari commented on PR #22493: URL: https://github.com/apache/pulsar/pull/22493#issuecomment-2051855377 > I remember your mentioned the similar issue in my PR before but I still didn't get in which case `BeforeClass` method is not called. Yes, there is something odd here since I wouldn't expect this class to be part of flaky tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][test] Fix NPE in BookKeeperClusterTestCase tearDown [pulsar]
BewareMyPower commented on PR #22493: URL: https://github.com/apache/pulsar/pull/22493#issuecomment-2051839540 I remember your mentioned the similar issue in my PR before but I still didn't get in which case `BeforeClass` method is not called. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch master updated (7984cc2f93f -> dbe1a4816c1)
This is an automated email from the ASF dual-hosted git repository. xyz pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git from 7984cc2f93f [feat][admin] Enable Gzip Compression by Default in Admin Client (#22464) add dbe1a4816c1 [improve][broker] Reduce the duplicated null check for LeaderElectionService (#22465) No new revisions were added by this update. Summary of changes: .../pulsar/broker/namespace/NamespaceService.java | 17 +++-- 1 file changed, 3 insertions(+), 14 deletions(-)
Re: [PR] [improve][broker] Reduce the duplicated null check for LeaderElectionService [pulsar]
BewareMyPower merged PR #22465: URL: https://github.com/apache/pulsar/pull/22465 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [D] Sending the message size greater then 5 MB from producer throwing error in pulsar [pulsar]
GitHub user slawrencemd added a comment to the discussion: Sending the message size greater then 5 MB from producer throwing error in pulsar Note: I'm pretty new to Pulsar myself, so this is just from my basic understanding. That error makes sense to me. Batching's purpose seems to be to be for increasing throughput gains - instead of making multiple TCP calls to the broker, use 1 network call with a batch of messages. It amortizes the network costs. I think you might be looking for [chunking ](https://pulsar.apache.org/docs/next/concepts-messaging/#chunking)instead. But note that chunking cannot be enabled when batching is on.  GitHub link: https://github.com/apache/pulsar/discussions/22492#discussioncomment-9096348 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
Re: [PR] [fix][broker] Check the broker is available for the SLA monitor bundle when the ExtensibleLoadManager is enabled [pulsar]
BewareMyPower commented on code in PR #22485:
URL: https://github.com/apache/pulsar/pull/22485#discussion_r1562531900
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImpl.java:
##
@@ -496,30 +496,42 @@ public CompletableFuture>
assign(Optional {
+if (candidateBrokerId != null) {
+return
CompletableFuture.completedFuture(Optional.of(candidateBrokerId));
+}
+return getOrSelectOwnerAsync(serviceUnit,
bundle).thenApply(Optional::ofNullable);
+});
}
return getBrokerLookupData(owner, bundle);
});
}
-private String getHeartbeatOrSLAMonitorBrokerId(ServiceUnitId serviceUnit)
{
+private CompletableFuture
getHeartbeatOrSLAMonitorBrokerId(ServiceUnitId serviceUnit) {
Review Comment:
You can just keep the synchronous way by calling `get` with the configured
metadata timeout.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
(pulsar) branch master updated: [feat][admin] Enable Gzip Compression by Default in Admin Client (#22464)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 7984cc2f93f [feat][admin] Enable Gzip Compression by Default in Admin
Client (#22464)
7984cc2f93f is described below
commit 7984cc2f93f8dc85b598ded1167508eae4ee06ec
Author: Lari Hotari
AuthorDate: Fri Apr 12 05:56:55 2024 -0700
[feat][admin] Enable Gzip Compression by Default in Admin Client (#22464)
---
.../pulsar/client/admin/PulsarAdminBuilder.java| 11 +-
pulsar-client-admin/pom.xml| 7 ++
.../admin/internal/PulsarAdminBuilderImpl.java | 22 +++-
.../client/admin/internal/PulsarAdminImpl.java | 8 +-
.../admin/internal/http/AsyncHttpConnector.java| 14 ++-
.../internal/http/AsyncHttpConnectorProvider.java | 9 +-
.../client/admin/internal/PulsarAdminGzipTest.java | 122 +
7 files changed, 183 insertions(+), 10 deletions(-)
diff --git
a/pulsar-client-admin-api/src/main/java/org/apache/pulsar/client/admin/PulsarAdminBuilder.java
b/pulsar-client-admin-api/src/main/java/org/apache/pulsar/client/admin/PulsarAdminBuilder.java
index 1260555a7c4..1b025a752d9 100644
---
a/pulsar-client-admin-api/src/main/java/org/apache/pulsar/client/admin/PulsarAdminBuilder.java
+++
b/pulsar-client-admin-api/src/main/java/org/apache/pulsar/client/admin/PulsarAdminBuilder.java
@@ -327,4 +327,13 @@ public interface PulsarAdminBuilder {
*/
PulsarAdminBuilder setContextClassLoader(ClassLoader
clientBuilderClassLoader);
-}
+/**
+ * Determines whether to include the "Accept-Encoding: gzip" header in
HTTP requests.
+ * By default, the "Accept-Encoding: gzip" header is included in HTTP
requests.
+ * If this is set to false, the "Accept-Encoding: gzip" header will not be
included in the requests.
+ *
+ * @param acceptGzipCompression A flag that indicates whether to include
the "Accept-Encoding: gzip" header in HTTP
+ * requests
+ */
+PulsarAdminBuilder acceptGzipCompression(boolean acceptGzipCompression);
+}
\ No newline at end of file
diff --git a/pulsar-client-admin/pom.xml b/pulsar-client-admin/pom.xml
index a8e4823f9f6..657be0513e5 100644
--- a/pulsar-client-admin/pom.xml
+++ b/pulsar-client-admin/pom.xml
@@ -113,6 +113,13 @@
hamcrest
test
+
+
+ com.github.tomakehurst
+ wiremock-jre8
+ ${wiremock.version}
+ test
+
diff --git
a/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/internal/PulsarAdminBuilderImpl.java
b/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/internal/PulsarAdminBuilderImpl.java
index 009fa67fbaa..f7b1695f5f3 100644
---
a/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/internal/PulsarAdminBuilderImpl.java
+++
b/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/internal/PulsarAdminBuilderImpl.java
@@ -38,10 +38,11 @@ public class PulsarAdminBuilderImpl implements
PulsarAdminBuilder {
protected ClientConfigurationData conf;
private ClassLoader clientBuilderClassLoader = null;
+private boolean acceptGzipCompression = true;
@Override
public PulsarAdmin build() throws PulsarClientException {
-return new PulsarAdminImpl(conf.getServiceUrl(), conf,
clientBuilderClassLoader);
+return new PulsarAdminImpl(conf.getServiceUrl(), conf,
clientBuilderClassLoader, acceptGzipCompression);
}
public PulsarAdminBuilderImpl() {
@@ -54,13 +55,24 @@ public class PulsarAdminBuilderImpl implements
PulsarAdminBuilder {
@Override
public PulsarAdminBuilder clone() {
-return new PulsarAdminBuilderImpl(conf.clone());
+PulsarAdminBuilderImpl pulsarAdminBuilder = new
PulsarAdminBuilderImpl(conf.clone());
+pulsarAdminBuilder.clientBuilderClassLoader = clientBuilderClassLoader;
+pulsarAdminBuilder.acceptGzipCompression = acceptGzipCompression;
+return pulsarAdminBuilder;
}
@Override
public PulsarAdminBuilder loadConf(Map config) {
conf = ConfigurationDataUtils.loadData(config, conf,
ClientConfigurationData.class);
setAuthenticationFromPropsIfAvailable(conf);
+if (config.containsKey("acceptGzipCompression")) {
+Object acceptGzipCompressionObj =
config.get("acceptGzipCompression");
+if (acceptGzipCompressionObj instanceof Boolean) {
+acceptGzipCompression = (Boolean) acceptGzipCompressionObj;
+} else {
+acceptGzipCompression =
Boolean.parseBoolean(acceptGzipCompressionObj.toString());
+}
+}
return this;
}
@@ -227,4 +239,10 @@ public class PulsarAdminBuilderImpl implements
PulsarAdminBuilder {
this.clientBuilderClassLoader =
Re: [PR] [feat][admin] Enable Gzip Compression by Default in Admin Client [pulsar]
lhotari merged PR #22464: URL: https://github.com/apache/pulsar/pull/22464 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [fix][test] Fix NPE in BookKeeperClusterTestCase tearDown [pulsar]
lhotari opened a new pull request, #22493: URL: https://github.com/apache/pulsar/pull/22493 ### Motivation - impacts ManagedLedgerBkTest.tearDown and ManagedLedgerFactoryChangeLedgerPathTest.tearDown ``` Error: Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.278 s <<< FAILURE! - in org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryChangeLedgerPathTest Error: org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryChangeLedgerPathTest.tearDown Time elapsed: 0.186 s <<< FAILURE! java.lang.NullPointerException: Cannot invoke "java.util.concurrent.ExecutorService.shutdownNow()" because "this.executor" is null at org.apache.bookkeeper.test.BookKeeperClusterTestCase.tearDown(BookKeeperClusterTestCase.java:225) ``` and ``` Error: Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.573 s <<< FAILURE! - in org.apache.bookkeeper.mledger.impl.ManagedLedgerBkTest Error: org.apache.bookkeeper.mledger.impl.ManagedLedgerBkTest.tearDown Time elapsed: 0.446 s <<< FAILURE! java.lang.NullPointerException: Cannot invoke "java.util.concurrent.ExecutorService.shutdownNow()" because "this.executor" is null at org.apache.bookkeeper.test.BookKeeperClusterTestCase.tearDown(BookKeeperClusterTestCase.java:225) ``` ### Modifications - add null check - add `alwaysRun=true` to Before* methods ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [D] How to validate chunking in consumer side [pulsar]
GitHub user ragaur-tibco edited a discussion: How to validate chunking in
consumer side
How can we validate or any use-case to check chunking is working in consumer
side
```
ConsumerBuilder consumerBuilder = client.newConsumer()
.topic("chunkTopic")
.subscriptionName("")
.maxPendingChunkedMessage(2)
.autoAckOldestChunkedMessageOnQueueFull(true)
.expireTimeOfIncompleteChunkedMessage(6, TimeUnit.MILLISECONDS)
.subscriptionType("Shared");
Consumer consumer= consumerBuilder.subscribe();
```
As mentioned in pulsar documentation
**maxPendingChunkedMessage** = We can limit the maximum number of chunked
messages a consumer maintains concurrently by configuring the
maxPendingChunkedMessage parameter.
**autoAckOldestChunkedMessageOnQueueFull** = When @maxPendingChunkedMessage
threshold provided and the consumer reaches this threshold, it drops the
outstanding unchunked-messages by silently acknowledging if
autoAckOldestChunkedMessageOnQueueFull is true
**expireTimeOfIncompleteChunkedMessage** = If the producer fails to publish all
the chunks of a message, then the consumer can expire incomplete chunks if the
consumer doesn't receive all chunks during the expiration period
But want to know if there is any use-case or any debug logs from pulsar side to
check and validate chunking parameters maxPendingChunkedMessage,
autoAckOldestChunkedMessageOnQueueFull and expireTimeOfIncompleteChunkedMessage
if I set the **maxPendingChunkedMessage**=2 and enable
**autoAckOldestChunkedMessageOnQueueFull** in consumer side, and send message
from producer of size 1000bytes and configure the max chunk size to 10 bytes
now how to check and validate or any debug logs for the
maxPendingChunkedMessage, autoAckOldestChunkedMessageOnQueueFull and
maxPendingChunkedMessage
CC: @lhotari
GitHub link: https://github.com/apache/pulsar/discussions/22491
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
Re: [D] How to validate chunking in consumer side [pulsar]
GitHub user ragaur-tibco edited a discussion: How to validate chunking in
consumer side
How can we validate or any use-case to check chunking is working in consumer
side
```
ConsumerBuilder consumerBuilder = client.newConsumer()
.topic("chunkTopic")
.subscriptionName("")
.maxPendingChunkedMessage(2)
.autoAckOldestChunkedMessageOnQueueFull(true)
.expireTimeOfIncompleteChunkedMessage(6, TimeUnit.MILLISECONDS)
.subscriptionType("Shared");
Consumer consumer= consumerBuilder.subscribe();
```
want to know if there is any use-case or any debug logs from pulsar side to
check and validate chunking parameters maxPendingChunkedMessage,
autoAckOldestChunkedMessageOnQueueFull and expireTimeOfIncompleteChunkedMessage
CC: @lhotari
GitHub link: https://github.com/apache/pulsar/discussions/22491
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
Re: [D] How to validate chunking in consumer side [pulsar]
GitHub user ragaur-tibco edited a discussion: How to validate chunking in
consumer side
How can we validate or any use-case to check chunking is working in consumer
side
```
ConsumerBuilder consumerBuilder = client.newConsumer()
.topic("chunkTopic")
.subscriptionName("")
.maxPendingChunkedMessage(2)
.autoAckOldestChunkedMessageOnQueueFull(true)
.expireTimeOfIncompleteChunkedMessage(6, TimeUnit.MILLISECONDS)
.subscriptionType("Shared");
Consumer consumer= consumerBuilder.subscribe();
```
want to know if there is any use-case or any debug logs from pulsar side to
check and validate chunking parameters maxPendingChunkedMessage,
autoAckOldestChunkedMessageOnQueueFull and expireTimeOfIncompleteChunkedMessage
GitHub link: https://github.com/apache/pulsar/discussions/22491
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
(pulsar) branch branch-3.1 updated: [improve][misc] Upgrade to Bookkeeper 4.16.5 (#22484)
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a commit to branch branch-3.1 in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/branch-3.1 by this push: new d975ac0656b [improve][misc] Upgrade to Bookkeeper 4.16.5 (#22484) d975ac0656b is described below commit d975ac0656b670cebbe2ec594409e6f64b909dbf Author: Lari Hotari AuthorDate: Fri Apr 12 00:15:33 2024 -0700 [improve][misc] Upgrade to Bookkeeper 4.16.5 (#22484) (cherry picked from commit 4a5400f0c66dab2c3fbb0050c8f537952fef1951) --- distribution/server/src/assemble/LICENSE.bin.txt | 56 distribution/shell/src/assemble/LICENSE.bin.txt | 6 +-- pom.xml | 2 +- pulsar-sql/presto-distribution/LICENSE | 30 ++--- 4 files changed, 47 insertions(+), 47 deletions(-) diff --git a/distribution/server/src/assemble/LICENSE.bin.txt b/distribution/server/src/assemble/LICENSE.bin.txt index 6eeffcebc43..f0a9c094c8a 100644 --- a/distribution/server/src/assemble/LICENSE.bin.txt +++ b/distribution/server/src/assemble/LICENSE.bin.txt @@ -345,34 +345,34 @@ The Apache Software License, Version 2.0 - net.java.dev.jna-jna-jpms-5.12.1.jar - net.java.dev.jna-jna-platform-jpms-5.12.1.jar * BookKeeper -- org.apache.bookkeeper-bookkeeper-common-4.16.4.jar -- org.apache.bookkeeper-bookkeeper-common-allocator-4.16.4.jar -- org.apache.bookkeeper-bookkeeper-proto-4.16.4.jar -- org.apache.bookkeeper-bookkeeper-server-4.16.4.jar -- org.apache.bookkeeper-bookkeeper-tools-framework-4.16.4.jar -- org.apache.bookkeeper-circe-checksum-4.16.4.jar -- org.apache.bookkeeper-cpu-affinity-4.16.4.jar -- org.apache.bookkeeper-statelib-4.16.4.jar -- org.apache.bookkeeper-stream-storage-api-4.16.4.jar -- org.apache.bookkeeper-stream-storage-common-4.16.4.jar -- org.apache.bookkeeper-stream-storage-java-client-4.16.4.jar -- org.apache.bookkeeper-stream-storage-java-client-base-4.16.4.jar -- org.apache.bookkeeper-stream-storage-proto-4.16.4.jar -- org.apache.bookkeeper-stream-storage-server-4.16.4.jar -- org.apache.bookkeeper-stream-storage-service-api-4.16.4.jar -- org.apache.bookkeeper-stream-storage-service-impl-4.16.4.jar -- org.apache.bookkeeper.http-http-server-4.16.4.jar -- org.apache.bookkeeper.http-vertx-http-server-4.16.4.jar -- org.apache.bookkeeper.stats-bookkeeper-stats-api-4.16.4.jar -- org.apache.bookkeeper.stats-prometheus-metrics-provider-4.16.4.jar -- org.apache.distributedlog-distributedlog-common-4.16.4.jar -- org.apache.distributedlog-distributedlog-core-4.16.4-tests.jar -- org.apache.distributedlog-distributedlog-core-4.16.4.jar -- org.apache.distributedlog-distributedlog-protocol-4.16.4.jar -- org.apache.bookkeeper.stats-codahale-metrics-provider-4.16.4.jar -- org.apache.bookkeeper-bookkeeper-slogger-api-4.16.4.jar -- org.apache.bookkeeper-bookkeeper-slogger-slf4j-4.16.4.jar -- org.apache.bookkeeper-native-io-4.16.4.jar +- org.apache.bookkeeper-bookkeeper-common-4.16.5.jar +- org.apache.bookkeeper-bookkeeper-common-allocator-4.16.5.jar +- org.apache.bookkeeper-bookkeeper-proto-4.16.5.jar +- org.apache.bookkeeper-bookkeeper-server-4.16.5.jar +- org.apache.bookkeeper-bookkeeper-tools-framework-4.16.5.jar +- org.apache.bookkeeper-circe-checksum-4.16.5.jar +- org.apache.bookkeeper-cpu-affinity-4.16.5.jar +- org.apache.bookkeeper-statelib-4.16.5.jar +- org.apache.bookkeeper-stream-storage-api-4.16.5.jar +- org.apache.bookkeeper-stream-storage-common-4.16.5.jar +- org.apache.bookkeeper-stream-storage-java-client-4.16.5.jar +- org.apache.bookkeeper-stream-storage-java-client-base-4.16.5.jar +- org.apache.bookkeeper-stream-storage-proto-4.16.5.jar +- org.apache.bookkeeper-stream-storage-server-4.16.5.jar +- org.apache.bookkeeper-stream-storage-service-api-4.16.5.jar +- org.apache.bookkeeper-stream-storage-service-impl-4.16.5.jar +- org.apache.bookkeeper.http-http-server-4.16.5.jar +- org.apache.bookkeeper.http-vertx-http-server-4.16.5.jar +- org.apache.bookkeeper.stats-bookkeeper-stats-api-4.16.5.jar +- org.apache.bookkeeper.stats-prometheus-metrics-provider-4.16.5.jar +- org.apache.distributedlog-distributedlog-common-4.16.5.jar +- org.apache.distributedlog-distributedlog-core-4.16.5-tests.jar +- org.apache.distributedlog-distributedlog-core-4.16.5.jar +- org.apache.distributedlog-distributedlog-protocol-4.16.5.jar +- org.apache.bookkeeper.stats-codahale-metrics-provider-4.16.5.jar +- org.apache.bookkeeper-bookkeeper-slogger-api-4.16.5.jar +- org.apache.bookkeeper-bookkeeper-slogger-slf4j-4.16.5.jar +- org.apache.bookkeeper-native-io-4.16.5.jar * Apache HTTP Client - org.apache.httpcomponents-httpclient-4.5.13.jar -
(pulsar) branch branch-3.2 updated: [improve][misc] Upgrade to Bookkeeper 4.16.5 (#22484)
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a commit to branch branch-3.2 in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/branch-3.2 by this push: new a2e1c73c766 [improve][misc] Upgrade to Bookkeeper 4.16.5 (#22484) a2e1c73c766 is described below commit a2e1c73c7665162ab7e808bff290b8b1f102e44b Author: Lari Hotari AuthorDate: Fri Apr 12 00:15:33 2024 -0700 [improve][misc] Upgrade to Bookkeeper 4.16.5 (#22484) (cherry picked from commit 4a5400f0c66dab2c3fbb0050c8f537952fef1951) --- distribution/server/src/assemble/LICENSE.bin.txt | 56 distribution/shell/src/assemble/LICENSE.bin.txt | 6 +-- pom.xml | 2 +- 3 files changed, 32 insertions(+), 32 deletions(-) diff --git a/distribution/server/src/assemble/LICENSE.bin.txt b/distribution/server/src/assemble/LICENSE.bin.txt index 2d0b4c5026e..f3e113a14d5 100644 --- a/distribution/server/src/assemble/LICENSE.bin.txt +++ b/distribution/server/src/assemble/LICENSE.bin.txt @@ -346,34 +346,34 @@ The Apache Software License, Version 2.0 - net.java.dev.jna-jna-jpms-5.12.1.jar - net.java.dev.jna-jna-platform-jpms-5.12.1.jar * BookKeeper -- org.apache.bookkeeper-bookkeeper-common-4.16.4.jar -- org.apache.bookkeeper-bookkeeper-common-allocator-4.16.4.jar -- org.apache.bookkeeper-bookkeeper-proto-4.16.4.jar -- org.apache.bookkeeper-bookkeeper-server-4.16.4.jar -- org.apache.bookkeeper-bookkeeper-tools-framework-4.16.4.jar -- org.apache.bookkeeper-circe-checksum-4.16.4.jar -- org.apache.bookkeeper-cpu-affinity-4.16.4.jar -- org.apache.bookkeeper-statelib-4.16.4.jar -- org.apache.bookkeeper-stream-storage-api-4.16.4.jar -- org.apache.bookkeeper-stream-storage-common-4.16.4.jar -- org.apache.bookkeeper-stream-storage-java-client-4.16.4.jar -- org.apache.bookkeeper-stream-storage-java-client-base-4.16.4.jar -- org.apache.bookkeeper-stream-storage-proto-4.16.4.jar -- org.apache.bookkeeper-stream-storage-server-4.16.4.jar -- org.apache.bookkeeper-stream-storage-service-api-4.16.4.jar -- org.apache.bookkeeper-stream-storage-service-impl-4.16.4.jar -- org.apache.bookkeeper.http-http-server-4.16.4.jar -- org.apache.bookkeeper.http-vertx-http-server-4.16.4.jar -- org.apache.bookkeeper.stats-bookkeeper-stats-api-4.16.4.jar -- org.apache.bookkeeper.stats-prometheus-metrics-provider-4.16.4.jar -- org.apache.distributedlog-distributedlog-common-4.16.4.jar -- org.apache.distributedlog-distributedlog-core-4.16.4-tests.jar -- org.apache.distributedlog-distributedlog-core-4.16.4.jar -- org.apache.distributedlog-distributedlog-protocol-4.16.4.jar -- org.apache.bookkeeper.stats-codahale-metrics-provider-4.16.4.jar -- org.apache.bookkeeper-bookkeeper-slogger-api-4.16.4.jar -- org.apache.bookkeeper-bookkeeper-slogger-slf4j-4.16.4.jar -- org.apache.bookkeeper-native-io-4.16.4.jar +- org.apache.bookkeeper-bookkeeper-common-4.16.5.jar +- org.apache.bookkeeper-bookkeeper-common-allocator-4.16.5.jar +- org.apache.bookkeeper-bookkeeper-proto-4.16.5.jar +- org.apache.bookkeeper-bookkeeper-server-4.16.5.jar +- org.apache.bookkeeper-bookkeeper-tools-framework-4.16.5.jar +- org.apache.bookkeeper-circe-checksum-4.16.5.jar +- org.apache.bookkeeper-cpu-affinity-4.16.5.jar +- org.apache.bookkeeper-statelib-4.16.5.jar +- org.apache.bookkeeper-stream-storage-api-4.16.5.jar +- org.apache.bookkeeper-stream-storage-common-4.16.5.jar +- org.apache.bookkeeper-stream-storage-java-client-4.16.5.jar +- org.apache.bookkeeper-stream-storage-java-client-base-4.16.5.jar +- org.apache.bookkeeper-stream-storage-proto-4.16.5.jar +- org.apache.bookkeeper-stream-storage-server-4.16.5.jar +- org.apache.bookkeeper-stream-storage-service-api-4.16.5.jar +- org.apache.bookkeeper-stream-storage-service-impl-4.16.5.jar +- org.apache.bookkeeper.http-http-server-4.16.5.jar +- org.apache.bookkeeper.http-vertx-http-server-4.16.5.jar +- org.apache.bookkeeper.stats-bookkeeper-stats-api-4.16.5.jar +- org.apache.bookkeeper.stats-prometheus-metrics-provider-4.16.5.jar +- org.apache.distributedlog-distributedlog-common-4.16.5.jar +- org.apache.distributedlog-distributedlog-core-4.16.5-tests.jar +- org.apache.distributedlog-distributedlog-core-4.16.5.jar +- org.apache.distributedlog-distributedlog-protocol-4.16.5.jar +- org.apache.bookkeeper.stats-codahale-metrics-provider-4.16.5.jar +- org.apache.bookkeeper-bookkeeper-slogger-api-4.16.5.jar +- org.apache.bookkeeper-bookkeeper-slogger-slf4j-4.16.5.jar +- org.apache.bookkeeper-native-io-4.16.5.jar * Apache HTTP Client - org.apache.httpcomponents-httpclient-4.5.13.jar - org.apache.httpcomponents-httpcore-4.4.15.jar diff --git
Re: [PR] [fix][test] Flaky-test: PersistentMessageFinderTest.testMessageExpiryWithTimestampNonRecoverableException [pulsar]
shibd commented on code in PR #22489:
URL: https://github.com/apache/pulsar/pull/22489#discussion_r1562366276
##
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/PersistentMessageFinderTest.java:
##
@@ -420,19 +423,13 @@ void
testMessageExpiryWithTimestampNonRecoverableException() throws Exception {
when(mock.getLastPosition()).thenReturn(PositionImpl.EARLIEST);
PersistentMessageExpiryMonitor monitor = new
PersistentMessageExpiryMonitor(mock, c1.getName(), c1, null);
-Position previousMarkDelete = null;
-for (int i = 0; i < totalEntries; i++) {
-monitor.expireMessages(1);
Review Comment:
The original code didn't make sense to me. Here just need to call
`monitor.expireMessages` once and then use Awaitility to assert
markDeletePositiion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [fix][test] Flaky-test: PersistentMessageFinderTest.testMessageExpiryWithTimestampNonRecoverableException [pulsar]
shibd commented on code in PR #22489:
URL: https://github.com/apache/pulsar/pull/22489#discussion_r1562366276
##
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/PersistentMessageFinderTest.java:
##
@@ -420,19 +423,13 @@ void
testMessageExpiryWithTimestampNonRecoverableException() throws Exception {
when(mock.getLastPosition()).thenReturn(PositionImpl.EARLIEST);
PersistentMessageExpiryMonitor monitor = new
PersistentMessageExpiryMonitor(mock, c1.getName(), c1, null);
-Position previousMarkDelete = null;
-for (int i = 0; i < totalEntries; i++) {
-monitor.expireMessages(1);
Review Comment:
Just need to call `monitor.expireMessages` once and then use Awaitility to
assert markDeletePositiion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
(pulsar) branch branch-3.0 updated: [improve][misc] Upgrade to Bookkeeper 4.16.5 (#22484)
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new 5ba3e5713e4 [improve][misc] Upgrade to Bookkeeper 4.16.5 (#22484) 5ba3e5713e4 is described below commit 5ba3e5713e47214d5ce2bb2591f483ee395d0263 Author: Lari Hotari AuthorDate: Fri Apr 12 00:15:33 2024 -0700 [improve][misc] Upgrade to Bookkeeper 4.16.5 (#22484) (cherry picked from commit 4a5400f0c66dab2c3fbb0050c8f537952fef1951) --- distribution/server/src/assemble/LICENSE.bin.txt | 56 distribution/shell/src/assemble/LICENSE.bin.txt | 6 +-- pom.xml | 2 +- pulsar-sql/presto-distribution/LICENSE | 30 ++--- 4 files changed, 47 insertions(+), 47 deletions(-) diff --git a/distribution/server/src/assemble/LICENSE.bin.txt b/distribution/server/src/assemble/LICENSE.bin.txt index 2720ebfe708..4a93cdfe2b7 100644 --- a/distribution/server/src/assemble/LICENSE.bin.txt +++ b/distribution/server/src/assemble/LICENSE.bin.txt @@ -346,34 +346,34 @@ The Apache Software License, Version 2.0 - net.java.dev.jna-jna-jpms-5.12.1.jar - net.java.dev.jna-jna-platform-jpms-5.12.1.jar * BookKeeper -- org.apache.bookkeeper-bookkeeper-common-4.16.4.jar -- org.apache.bookkeeper-bookkeeper-common-allocator-4.16.4.jar -- org.apache.bookkeeper-bookkeeper-proto-4.16.4.jar -- org.apache.bookkeeper-bookkeeper-server-4.16.4.jar -- org.apache.bookkeeper-bookkeeper-tools-framework-4.16.4.jar -- org.apache.bookkeeper-circe-checksum-4.16.4.jar -- org.apache.bookkeeper-cpu-affinity-4.16.4.jar -- org.apache.bookkeeper-statelib-4.16.4.jar -- org.apache.bookkeeper-stream-storage-api-4.16.4.jar -- org.apache.bookkeeper-stream-storage-common-4.16.4.jar -- org.apache.bookkeeper-stream-storage-java-client-4.16.4.jar -- org.apache.bookkeeper-stream-storage-java-client-base-4.16.4.jar -- org.apache.bookkeeper-stream-storage-proto-4.16.4.jar -- org.apache.bookkeeper-stream-storage-server-4.16.4.jar -- org.apache.bookkeeper-stream-storage-service-api-4.16.4.jar -- org.apache.bookkeeper-stream-storage-service-impl-4.16.4.jar -- org.apache.bookkeeper.http-http-server-4.16.4.jar -- org.apache.bookkeeper.http-vertx-http-server-4.16.4.jar -- org.apache.bookkeeper.stats-bookkeeper-stats-api-4.16.4.jar -- org.apache.bookkeeper.stats-prometheus-metrics-provider-4.16.4.jar -- org.apache.distributedlog-distributedlog-common-4.16.4.jar -- org.apache.distributedlog-distributedlog-core-4.16.4-tests.jar -- org.apache.distributedlog-distributedlog-core-4.16.4.jar -- org.apache.distributedlog-distributedlog-protocol-4.16.4.jar -- org.apache.bookkeeper.stats-codahale-metrics-provider-4.16.4.jar -- org.apache.bookkeeper-bookkeeper-slogger-api-4.16.4.jar -- org.apache.bookkeeper-bookkeeper-slogger-slf4j-4.16.4.jar -- org.apache.bookkeeper-native-io-4.16.4.jar +- org.apache.bookkeeper-bookkeeper-common-4.16.5.jar +- org.apache.bookkeeper-bookkeeper-common-allocator-4.16.5.jar +- org.apache.bookkeeper-bookkeeper-proto-4.16.5.jar +- org.apache.bookkeeper-bookkeeper-server-4.16.5.jar +- org.apache.bookkeeper-bookkeeper-tools-framework-4.16.5.jar +- org.apache.bookkeeper-circe-checksum-4.16.5.jar +- org.apache.bookkeeper-cpu-affinity-4.16.5.jar +- org.apache.bookkeeper-statelib-4.16.5.jar +- org.apache.bookkeeper-stream-storage-api-4.16.5.jar +- org.apache.bookkeeper-stream-storage-common-4.16.5.jar +- org.apache.bookkeeper-stream-storage-java-client-4.16.5.jar +- org.apache.bookkeeper-stream-storage-java-client-base-4.16.5.jar +- org.apache.bookkeeper-stream-storage-proto-4.16.5.jar +- org.apache.bookkeeper-stream-storage-server-4.16.5.jar +- org.apache.bookkeeper-stream-storage-service-api-4.16.5.jar +- org.apache.bookkeeper-stream-storage-service-impl-4.16.5.jar +- org.apache.bookkeeper.http-http-server-4.16.5.jar +- org.apache.bookkeeper.http-vertx-http-server-4.16.5.jar +- org.apache.bookkeeper.stats-bookkeeper-stats-api-4.16.5.jar +- org.apache.bookkeeper.stats-prometheus-metrics-provider-4.16.5.jar +- org.apache.distributedlog-distributedlog-common-4.16.5.jar +- org.apache.distributedlog-distributedlog-core-4.16.5-tests.jar +- org.apache.distributedlog-distributedlog-core-4.16.5.jar +- org.apache.distributedlog-distributedlog-protocol-4.16.5.jar +- org.apache.bookkeeper.stats-codahale-metrics-provider-4.16.5.jar +- org.apache.bookkeeper-bookkeeper-slogger-api-4.16.5.jar +- org.apache.bookkeeper-bookkeeper-slogger-slf4j-4.16.5.jar +- org.apache.bookkeeper-native-io-4.16.5.jar * Apache HTTP Client - org.apache.httpcomponents-httpclient-4.5.13.jar -
Re: [PR] [fix] [broker] Part-1: Replicator can not created successfully due to an orphan replicator in the previous topic owner [pulsar]
gaoran10 commented on code in PR #21946:
URL: https://github.com/apache/pulsar/pull/21946#discussion_r1562306732
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractReplicator.java:
##
@@ -188,58 +270,135 @@ protected CompletableFuture
isLocalTopicActive() {
}, brokerService.executor());
}
-protected synchronized CompletableFuture closeProducerAsync() {
-if (producer == null) {
-STATE_UPDATER.set(this, State.Stopped);
+/**
+ * This method only be used by {@link PersistentTopic#checkGC} now.
+ */
+public CompletableFuture disconnect(boolean failIfHasBacklog,
boolean closeTheStartingProducer) {
+long backlog = getNumberOfEntriesInBacklog();
+if (failIfHasBacklog && backlog > 0) {
+CompletableFuture disconnectFuture = new
CompletableFuture<>();
+disconnectFuture.completeExceptionally(new
TopicBusyException("Cannot close a replicator with backlog"));
+if (log.isDebugEnabled()) {
+log.debug("[{}] Replicator disconnect failed since topic has
backlog", replicatorId);
+}
+return disconnectFuture;
+}
+log.info("[{}] Disconnect replicator at position {} with backlog {}",
replicatorId,
+getReplicatorReadPosition(), backlog);
+return closeProducerAsync(closeTheStartingProducer);
+}
+
+/**
+ * This method only be used by {@link PersistentTopic#checkGC} now.
+ */
+protected CompletableFuture closeProducerAsync(boolean
closeTheStartingProducer) {
+Pair setDisconnectingRes =
compareSetAndGetState(State.Started, State.Disconnecting);
+if (!setDisconnectingRes.getLeft()) {
+if (setDisconnectingRes.getRight() == State.Starting) {
+if (closeTheStartingProducer) {
+/**
+ * Delay retry(wait for the start producer task is finish).
+ * Note: If the producer always start fail, the start
producer task will always retry until the
+ * state changed to {@link State.Terminated}.
+ * Nit: The better solution is creating a {@link
CompletableFuture} to trace the in-progress
+ * creation and call
"inProgressCreationFuture.thenApply(closeProducer())".
+ */
+long waitTimeMs = backOff.next();
+brokerService.executor().schedule(() ->
closeProducerAsync(true),
+waitTimeMs, TimeUnit.MILLISECONDS);
+} else {
+log.info("[{}] Skip current producer closing since the
previous producer has been closed,"
++ " and trying start a new one, state :
{}",
+replicatorId, setDisconnectingRes.getRight());
+}
+} else if (setDisconnectingRes.getRight() == State.Disconnected
+|| setDisconnectingRes.getRight() == State.Disconnecting) {
+log.info("[{}] Skip current producer closing since other
thread did closing, state : {}",
+replicatorId, setDisconnectingRes.getRight());
+} else if (setDisconnectingRes.getRight() == State.Terminating
+|| setDisconnectingRes.getRight() == State.Terminated) {
+log.info("[{}] Skip current producer closing since other
thread is doing termination, state : {}",
+replicatorId, state);
+}
+log.info("[{}] Skip current termination since other thread is
doing close producer or termination,"
++ " state : {}", replicatorId, state);
return CompletableFuture.completedFuture(null);
}
-CompletableFuture future = producer.closeAsync();
+
+// Close producer and update state.
+return doCloseProducerAsync(producer, () -> {
+Pair setDisconnectedRes =
compareSetAndGetState(State.Disconnecting, State.Disconnected);
+if (setDisconnectedRes.getLeft()) {
+this.producer = null;
+// deactivate further read
+disableReplicatorRead();
+return;
+}
+if (setDisconnectedRes.getRight() == State.Terminating
+|| setDisconnectingRes.getRight() == State.Terminated) {
+log.info("[{}] Skip setting state to terminated because it was
terminated, state : {}",
+replicatorId, state);
+} else {
+// Since only one task can call
"doCloseProducerAsync(producer, action)", this scenario is not expected.
+// So print a warn log.
+log.warn("[{}] Other task has change the state to terminated.
so skipped current one task."
++ " State is : {}",
+
(pulsar-helm-chart) branch master updated: Fix prometheus node-exporter crashloop (#488)
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar-helm-chart.git The following commit(s) were added to refs/heads/master by this push: new 59f6f74 Fix prometheus node-exporter crashloop (#488) 59f6f74 is described below commit 59f6f74fd7e1adbabe41b7dd184c7f201525669f Author: Lari Hotari AuthorDate: Fri Apr 12 03:10:24 2024 -0700 Fix prometheus node-exporter crashloop (#488) --- charts/pulsar/values.yaml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/charts/pulsar/values.yaml b/charts/pulsar/values.yaml index 50fbc1c..1094510 100755 --- a/charts/pulsar/values.yaml +++ b/charts/pulsar/values.yaml @@ -1256,6 +1256,8 @@ kube-prometheus-stack: datasource: Prometheus prometheus-node-exporter: enabled: true +hostRootFsMount: + enabled: false alertmanager: enabled: false
Re: [PR] Fix prometheus node-exporter crashloop [pulsar-helm-chart]
lhotari merged PR #488: URL: https://github.com/apache/pulsar-helm-chart/pull/488 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] fix: Compatible with Node 12.3.0 and higher versions. [pulsar-client-node]
shibd opened a new pull request, #378:
URL: https://github.com/apache/pulsar-client-node/pull/378
### Motivation
-
https://github.com/apache/pulsar-client-node/issues/226#issuecomment-1917492231
- https://lists.apache.org/thread/6rmk2m09q1c4f2g3s8thf5fgd0vsfryv
After #301, will use the [Mozilla
CA](https://nodejs.org/api/tls.html#tlsrootcertificates) certificate provided
by the node.js. And use the `fs.rmSync` API to generate a local ca when install.
But the [fs.rmSync](https://nodejs.org/api/fs.html#fsrmsyncpath-options) was
introduced in Node.js v14.14.0, so an error will occur when installing now.
```
TypeError: fs.rmSync is not a function
```
### Modifications
- Use `fs.existsSync and unlinkSync ` instead of `fs.rmSync`.
- Since [Mozilla CA](https://nodejs.org/api/tls.html#tlsrootcertificates)
requires a Node.js version greater than `v12.3.0`, the `README` and
`package.json` should be modified to indicate that it is only compatible with
versions 12.3.0 and above.
### Verifying this change
Test on ubuntu-22.04
Use node.js v12.3.0 can work.
```shell
gitpod /workspace/pulsar-client-node (compatible_version) $ npm install
npm WARN read-shrinkwrap This version of npm is compatible with
lockfileVersion@1, but package-lock.json was generated for lockfileVersion@2.
I'll try to do my best with it!
> pulsar-client@1.12.0-rc.0 install /workspace/pulsar-client-node
> node-pre-gyp install --fallback-to-build && node GenCertFile.js
[pulsar-client] Success:
"/workspace/pulsar-client-node/lib/binding/pulsar.node" already installed
Pass --update-binary to reinstall or --build-from-source to recompile
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@2.3.2
(node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for
fsevents@2.3.2: wanted {"os":"darwin","arch":"any"} (current:
{"os":"linux","arch":"x64"})
added 746 packages from 559 contributors and audited 748 packages in 10.463s
found 6 moderate severity vulnerabilities
run `npm audit fix` to fix them, or `npm audit` for details
```
### Documentation
- [ ] `doc-required`
(Your PR needs to update docs and you will update later)
- [x] `doc-not-needed`
(Please explain why)
- [ ] `doc`
(Your PR contains doc changes)
- [ ] `doc-complete`
(Docs have been already added)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [improve][broker] Improve Gzip compression, allow excluding specific paths or disabling it [pulsar]
lhotari commented on code in PR #22370:
URL: https://github.com/apache/pulsar/pull/22370#discussion_r1562323798
##
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java:
##
@@ -331,6 +331,19 @@ public class ServiceConfiguration implements
PulsarConfiguration {
+ "(0 to disable limiting)")
private int maxHttpServerConnections = 2048;
+@FieldContext(category = CATEGORY_SERVER, doc =
Review Comment:
@hangc0276 I'd suggest a separate change for that since making this dynamic
will be a significant change to the current PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [fix][test] Flaky-test: PersistentMessageFinderTest.testMessageExpiryWithTimestampNonRecoverableException [pulsar]
lhotari commented on code in PR #22489:
URL: https://github.com/apache/pulsar/pull/22489#discussion_r1562320433
##
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/PersistentMessageFinderTest.java:
##
@@ -402,11 +402,11 @@ void
testMessageExpiryWithTimestampNonRecoverableException() throws Exception {
for (int i = 0; i < totalEntries; i++) {
ledger.addEntry(createMessageWrittenToLedger("msg" + i));
}
+retryStrategically((test) -> ledger.getState() ==
ManagedLedgerImpl.State.LedgerOpened,10, 200);
Review Comment:
`retryStrategically` should be replaced with Awaitility in most cases. I
guess `retryStrategically` was introduced to the code base before Awaitility
existed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [improve] Make the config `metricsBufferResponse` description more effective [pulsar]
lhotari commented on code in PR #22490: URL: https://github.com/apache/pulsar/pull/22490#discussion_r1562306801 ## pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java: ## @@ -2915,7 +2915,9 @@ The max allowed delay for delayed delivery (in milliseconds). If the broker rece private boolean exposeTopicLevelMetricsInPrometheus = true; @FieldContext( category = CATEGORY_METRICS, -doc = "If true, export buffered metrics" +doc = "Set to true to enable the broker to cache the metrics response, default is false. " ++ "For scraping metrics more than once per scrape period, the broker generates metrics at the first " ++ "scrape and returns the same response for the rest of the scrape period. " Review Comment: Add a better definition for scraping period. Currently "scrape period" is vague. It's better to explicitly define it. There's 2 different concepts that are involved: * Internally, there's a caching period for the metrics response. In the implementation, the caching period is defined with `managedLedgerStatsPeriodSeconds`. * In Prometheus, there's a global setting called ["scrape_interval"](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#configuration-file) and in the Prometheus operator there's [PodMetricsEndpoint interval](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#monitoring.coreos.com/v1.PodMetricsEndpoint) for configuring the scraping interval for individual scraping endpoints. Please revisit the description to resolve the issue with the undefined concept "scrape period". -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][broker] Check the broker is available for the SLA monitor bundle when the ExtensibleLoadManager is enabled [pulsar]
Demogorgon314 commented on code in PR #22485:
URL: https://github.com/apache/pulsar/pull/22485#discussion_r1562277230
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImpl.java:
##
@@ -496,30 +496,42 @@ public CompletableFuture>
assign(Optional {
+if (candidateBrokerId != null) {
+return
CompletableFuture.completedFuture(Optional.of(candidateBrokerId));
+}
+return getOrSelectOwnerAsync(serviceUnit,
bundle).thenApply(Optional::ofNullable);
+});
}
return getBrokerLookupData(owner, bundle);
});
}
-private String getHeartbeatOrSLAMonitorBrokerId(ServiceUnitId serviceUnit)
{
+private CompletableFuture
getHeartbeatOrSLAMonitorBrokerId(ServiceUnitId serviceUnit) {
Review Comment:
This change requires making the `NamespaceService#searchForCandidateBroker`
async and introducing many changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [fix][broker] Check the broker is available for the SLA monitor bundle when the ExtensibleLoadManager is enabled [pulsar]
BewareMyPower commented on code in PR #22485:
URL: https://github.com/apache/pulsar/pull/22485#discussion_r1562248320
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImpl.java:
##
@@ -496,30 +496,42 @@ public CompletableFuture>
assign(Optional {
+if (candidateBrokerId != null) {
+return
CompletableFuture.completedFuture(Optional.of(candidateBrokerId));
+}
+return getOrSelectOwnerAsync(serviceUnit,
bundle).thenApply(Optional::ofNullable);
+});
}
return getBrokerLookupData(owner, bundle);
});
}
-private String getHeartbeatOrSLAMonitorBrokerId(ServiceUnitId serviceUnit)
{
+private CompletableFuture
getHeartbeatOrSLAMonitorBrokerId(ServiceUnitId serviceUnit) {
// Check if this is Heartbeat or SLAMonitor namespace
String candidateBroker =
NamespaceService.checkHeartbeatNamespace(serviceUnit);
-if (candidateBroker == null) {
-candidateBroker =
NamespaceService.checkHeartbeatNamespaceV2(serviceUnit);
+if (candidateBroker != null) {
+return CompletableFuture.completedFuture(candidateBroker);
}
-if (candidateBroker == null) {
-candidateBroker =
NamespaceService.getSLAMonitorBrokerName(serviceUnit);
+candidateBroker =
NamespaceService.checkHeartbeatNamespaceV2(serviceUnit);
+if (candidateBroker != null) {
+return CompletableFuture.completedFuture(candidateBroker);
}
+candidateBroker =
NamespaceService.getSLAMonitorBrokerName(serviceUnit);
if (candidateBroker != null) {
-return candidateBroker.substring(candidateBroker.lastIndexOf('/')
+ 1);
+// Check if the broker is available
+final String finalCandidateBroker = candidateBroker;
+return
brokerRegistry.lookupAsync(candidateBroker).thenApply(brokerLookupData -> {
+if (brokerLookupData.isEmpty()) {
+if (debug(conf, log)) {
+log.info("The SLA Monitor broker {} is not
available.", finalCandidateBroker);
Review Comment:
I see the same pattern in other places (e.g. `createSystemTopic`), so mark
it as resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [fix][broker] Check the broker is available for the SLA monitor bundle when the ExtensibleLoadManager is enabled [pulsar]
BewareMyPower commented on code in PR #22485:
URL: https://github.com/apache/pulsar/pull/22485#discussion_r1562242272
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImpl.java:
##
@@ -496,30 +496,42 @@ public CompletableFuture>
assign(Optional {
+if (candidateBrokerId != null) {
+return
CompletableFuture.completedFuture(Optional.of(candidateBrokerId));
+}
+return getOrSelectOwnerAsync(serviceUnit,
bundle).thenApply(Optional::ofNullable);
+});
}
return getBrokerLookupData(owner, bundle);
});
}
-private String getHeartbeatOrSLAMonitorBrokerId(ServiceUnitId serviceUnit)
{
+private CompletableFuture
getHeartbeatOrSLAMonitorBrokerId(ServiceUnitId serviceUnit) {
Review Comment:
For example,
```java
public static CompletableFuture getHeartbeatOrSLAMonitorBrokerId(
ServiceUnitId serviceUnit, LockManager
brokerLookupData) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[PR] [improve] Make the config `metricsBufferResponse` description more effective [pulsar]
dao-jun opened a new pull request, #22490: URL: https://github.com/apache/pulsar/pull/22490 ### Motivation Referring to: https://github.com/apache/pulsar/issues/22477#issuecomment-2050141202 ### Modifications ### Verifying this change - [ ] Make sure that the change passes the CI checks. *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Does this pull request potentially affect one of the following parts: *If the box was checked, please highlight the changes* - [ ] Dependencies (add or upgrade a dependency) - [ ] The public API - [ ] The schema - [ ] The default values of configurations - [ ] The threading model - [ ] The binary protocol - [ ] The REST endpoints - [ ] The admin CLI options - [ ] The metrics - [ ] Anything that affects deployment ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` ### Matching PR in forked repository PR in forked repository: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][broker] Check the broker is available for the SLA monitor bundle when the ExtensibleLoadManager is enabled [pulsar]
BewareMyPower commented on code in PR #22485:
URL: https://github.com/apache/pulsar/pull/22485#discussion_r1562234371
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImpl.java:
##
@@ -496,30 +496,42 @@ public CompletableFuture>
assign(Optional {
+if (candidateBrokerId != null) {
+return
CompletableFuture.completedFuture(Optional.of(candidateBrokerId));
+}
+return getOrSelectOwnerAsync(serviceUnit,
bundle).thenApply(Optional::ofNullable);
+});
}
return getBrokerLookupData(owner, bundle);
});
}
-private String getHeartbeatOrSLAMonitorBrokerId(ServiceUnitId serviceUnit)
{
+private CompletableFuture
getHeartbeatOrSLAMonitorBrokerId(ServiceUnitId serviceUnit) {
Review Comment:
We can add a `LockManager` field to `NamespaceService` and
apply the similar logic of `BrokerRegistryImpl#lookupAsync` to check if the
broker is alive.
```java
this.brokerLookupDataLockManager =
pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
(pulsar-client-node) branch master updated: Bump tar from 6.1.15 to 6.2.1 (#377)
This is an automated email from the ASF dual-hosted git repository.
baodi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar-client-node.git
The following commit(s) were added to refs/heads/master by this push:
new 2ed4a1e Bump tar from 6.1.15 to 6.2.1 (#377)
2ed4a1e is described below
commit 2ed4a1eefe0749641d08b2bcd8069506bab82ad8
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
AuthorDate: Fri Apr 12 16:41:04 2024 +0800
Bump tar from 6.1.15 to 6.2.1 (#377)
Bumps [tar](https://github.com/isaacs/node-tar) from 6.1.15 to 6.2.1.
- [Release notes](https://github.com/isaacs/node-tar/releases)
- [Changelog](https://github.com/isaacs/node-tar/blob/main/CHANGELOG.md)
- [Commits](https://github.com/isaacs/node-tar/compare/v6.1.15...v6.2.1)
---
updated-dependencies:
- dependency-name: tar
dependency-type: indirect
...
Signed-off-by: dependabot[bot]
Co-authored-by: dependabot[bot]
<49699333+dependabot[bot]@users.noreply.github.com>
---
package-lock.json | 12 ++--
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/package-lock.json b/package-lock.json
index c0a2830..f0638c1 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -7822,9 +7822,9 @@
"dev": true
},
"node_modules/tar": {
- "version": "6.1.15",
- "resolved": "https://registry.npmjs.org/tar/-/tar-6.1.15.tgz;,
- "integrity":
"sha512-/zKt9UyngnxIT/EAGYuxaMYgOIJiP81ab9ZfkILq4oNLPFX50qyYmu7jRj9qeXoxmJHjGlbH0+cm2uy1WCs10A==",
+ "version": "6.2.1",
+ "resolved": "https://registry.npmjs.org/tar/-/tar-6.2.1.tgz;,
+ "integrity":
"sha512-DZ4yORTwrbTj/7MZYq2w+/ZFdI6OZ/f9SFHR+71gIVUZhOQPHzVCLpvRnPgyaMpfWxxk/4ONva3GQSyNIKRv6A==",
"dependencies": {
"chownr": "^2.0.0",
"fs-minipass": "^2.0.0",
@@ -14790,9 +14790,9 @@
}
},
"tar": {
- "version": "6.1.15",
- "resolved": "https://registry.npmjs.org/tar/-/tar-6.1.15.tgz;,
- "integrity":
"sha512-/zKt9UyngnxIT/EAGYuxaMYgOIJiP81ab9ZfkILq4oNLPFX50qyYmu7jRj9qeXoxmJHjGlbH0+cm2uy1WCs10A==",
+ "version": "6.2.1",
+ "resolved": "https://registry.npmjs.org/tar/-/tar-6.2.1.tgz;,
+ "integrity":
"sha512-DZ4yORTwrbTj/7MZYq2w+/ZFdI6OZ/f9SFHR+71gIVUZhOQPHzVCLpvRnPgyaMpfWxxk/4ONva3GQSyNIKRv6A==",
"requires": {
"chownr": "^2.0.0",
"fs-minipass": "^2.0.0",
(pulsar-client-node) branch dependabot/npm_and_yarn/tar-6.2.1 deleted (was 1904885)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/npm_and_yarn/tar-6.2.1 in repository https://gitbox.apache.org/repos/asf/pulsar-client-node.git was 1904885 Bump tar from 6.1.15 to 6.2.1 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
Re: [PR] Bump tar from 6.1.15 to 6.2.1 [pulsar-client-node]
shibd merged PR #377: URL: https://github.com/apache/pulsar-client-node/pull/377 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][broker] Check the broker is available for the SLA monitor bundle when the ExtensibleLoadManager is enabled [pulsar]
BewareMyPower commented on code in PR #22485:
URL: https://github.com/apache/pulsar/pull/22485#discussion_r1562229038
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImpl.java:
##
@@ -496,30 +496,42 @@ public CompletableFuture>
assign(Optional {
+if (candidateBrokerId != null) {
+return
CompletableFuture.completedFuture(Optional.of(candidateBrokerId));
+}
+return getOrSelectOwnerAsync(serviceUnit,
bundle).thenApply(Optional::ofNullable);
+});
}
return getBrokerLookupData(owner, bundle);
});
}
-private String getHeartbeatOrSLAMonitorBrokerId(ServiceUnitId serviceUnit)
{
+private CompletableFuture
getHeartbeatOrSLAMonitorBrokerId(ServiceUnitId serviceUnit) {
Review Comment:
Could we add a public method to `NamespaceService` and reuse that method so
that we can avoid duplicated logic here? I think
`NamespaceService#isBrokerActive` can be replaced by the similar logic because
it calls `LoadManager#getAvailableBrokers`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [fix][broker] Check the broker is available for the SLA monitor bundle when the ExtensibleLoadManager is enabled [pulsar]
BewareMyPower commented on code in PR #22485:
URL: https://github.com/apache/pulsar/pull/22485#discussion_r156460
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImpl.java:
##
@@ -496,30 +496,42 @@ public CompletableFuture>
assign(Optional {
+if (candidateBrokerId != null) {
+return
CompletableFuture.completedFuture(Optional.of(candidateBrokerId));
+}
+return getOrSelectOwnerAsync(serviceUnit,
bundle).thenApply(Optional::ofNullable);
+});
}
return getBrokerLookupData(owner, bundle);
});
}
-private String getHeartbeatOrSLAMonitorBrokerId(ServiceUnitId serviceUnit)
{
+private CompletableFuture
getHeartbeatOrSLAMonitorBrokerId(ServiceUnitId serviceUnit) {
// Check if this is Heartbeat or SLAMonitor namespace
String candidateBroker =
NamespaceService.checkHeartbeatNamespace(serviceUnit);
-if (candidateBroker == null) {
-candidateBroker =
NamespaceService.checkHeartbeatNamespaceV2(serviceUnit);
+if (candidateBroker != null) {
+return CompletableFuture.completedFuture(candidateBroker);
}
-if (candidateBroker == null) {
-candidateBroker =
NamespaceService.getSLAMonitorBrokerName(serviceUnit);
+candidateBroker =
NamespaceService.checkHeartbeatNamespaceV2(serviceUnit);
+if (candidateBroker != null) {
+return CompletableFuture.completedFuture(candidateBroker);
}
+candidateBroker =
NamespaceService.getSLAMonitorBrokerName(serviceUnit);
if (candidateBroker != null) {
-return candidateBroker.substring(candidateBroker.lastIndexOf('/')
+ 1);
+// Check if the broker is available
+final String finalCandidateBroker = candidateBroker;
+return
brokerRegistry.lookupAsync(candidateBroker).thenApply(brokerLookupData -> {
+if (brokerLookupData.isEmpty()) {
+if (debug(conf, log)) {
+log.info("The SLA Monitor broker {} is not
available.", finalCandidateBroker);
Review Comment:
```suggestion
log.debug("The SLA Monitor broker {} is not
available.", finalCandidateBroker);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[PR] [fix][test] Flaky-test: PersistentMessageFinderTest.testMessageExpiryWithTimestampNonRecoverableException [pulsar]
shibd opened a new pull request, #22489: URL: https://github.com/apache/pulsar/pull/22489 ### Motivation #22351 After #22034, the calculation formula for the ledger number should be: `totalEntries / entriesPerLedger + 1`, However, since the creation of the ledger is asynchronous, sometimes the last ledger is not created before the function returns. https://github.com/apache/pulsar/blob/57a616eaa79096af5b49db89c99cd39ccc94ec00/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L1775-L1778 ### Modifications - Add `retryStrategically((test) -> ledger.getState() == ManagedLedgerImpl.State.LedgerOpened,10, 200)` to make sure later ledger creates success. - Modify the ledger quantity and assertions related to `makePosition`. ### Verifying this change - After running it 100 times locally, this test is stable. ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` ### Matching PR in forked repository -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] /metrics endpoint processes requests one-by-one and seems to queue up waiting requests infinitely, ignoring the request timeout [pulsar]
dao-jun commented on issue #22477: URL: https://github.com/apache/pulsar/issues/22477#issuecomment-2051246330 > Very cool - I was not aware of that setting. That said the current description for that parameter isn't helpful in understanding just how useful it can be :-) https://pulsar.apache.org/reference/#/next/config/reference-configuration-broker?id=optional `If true, export buffered metrics` I'll improvement the description of the configuration field. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [improve][broker] backlog quota exceed limit log replaced with `debug` [pulsar]
mukesh154 opened a new pull request, #22488: URL: https://github.com/apache/pulsar/pull/22488 Fixes #22474 Already fixed on branch-3.1: #22475 ### Motivation Currently, when backlog quota limit exceeded and producers still send messages, it generates lot of logs in short period of time. The logs are logged with `info`. We can use `debug` instead of `info`. ### Modifications - Replaced the `info` with `debug` for logs when the backlog quota limit exceeded. ### Verifying this change This change is a rework/code-cleanup without any test coverage. ### Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API: no - The schema: no - The default values of configurations: no - The wire protocol: no - The rest endpoints: no - The admin cli options: no - Anything that affects deployment: no ### Documentation Check the box below or label this PR directly. Need to update docs? - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [fix][broker] Support lookup options for extensible load manager [pulsar]
Demogorgon314 opened a new pull request, #22487: URL: https://github.com/apache/pulsar/pull/22487 ### Motivation Support lookup options for extensible load manager to align with the old behavior. ### Modifications Support the following lookup options: * readOnly * requestHttps * advertisedListenerName ### TODO Support `LookupOptions.loadTopicsInBundle` ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch branch-3.2 updated: Remove unused fields `msgSize`
This is an automated email from the ASF dual-hosted git repository.
zhaocong pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new be9b97d1ccc Remove unused fields `msgSize`
be9b97d1ccc is described below
commit be9b97d1ccc1ee2485ad8579aa81a94542cc5b1f
Author: coderzc
AuthorDate: Fri Apr 12 15:35:56 2024 +0800
Remove unused fields `msgSize`
---
.../src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java | 2 --
1 file changed, 2 deletions(-)
diff --git
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
index 04dfb01eeaa..1d1df7942ab 100644
---
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
+++
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
@@ -347,7 +347,6 @@ public class ProducerImpl extends ProducerBase
implements TimerTask, Conne
CompletableFuture sendFuture;
MessageImpl currentMsg;
-int msgSize;
long createdAt = System.nanoTime();
SendCallback nextCallback = null;
MessageImpl nextMsg = null;
@@ -355,7 +354,6 @@ public class ProducerImpl extends ProducerBase
implements TimerTask, Conne
DefaultSendMessageCallback(CompletableFuture sendFuture,
MessageImpl currentMsg, int msgSize) {
this.sendFuture = sendFuture;
this.currentMsg = currentMsg;
-this.msgSize = msgSize;
}
@Override
Re: [PR] [improve][misc] Upgrade to Bookkeeper 4.16.5 [pulsar]
lhotari merged PR #22484: URL: https://github.com/apache/pulsar/pull/22484 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch master updated: [improve][misc] Upgrade to Bookkeeper 4.16.5 (#22484)
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new 4a5400f0c66 [improve][misc] Upgrade to Bookkeeper 4.16.5 (#22484) 4a5400f0c66 is described below commit 4a5400f0c66dab2c3fbb0050c8f537952fef1951 Author: Lari Hotari AuthorDate: Fri Apr 12 00:15:33 2024 -0700 [improve][misc] Upgrade to Bookkeeper 4.16.5 (#22484) --- distribution/server/src/assemble/LICENSE.bin.txt | 56 distribution/shell/src/assemble/LICENSE.bin.txt | 6 +-- pom.xml | 2 +- 3 files changed, 32 insertions(+), 32 deletions(-) diff --git a/distribution/server/src/assemble/LICENSE.bin.txt b/distribution/server/src/assemble/LICENSE.bin.txt index 518f9231375..a409ad07ed1 100644 --- a/distribution/server/src/assemble/LICENSE.bin.txt +++ b/distribution/server/src/assemble/LICENSE.bin.txt @@ -356,34 +356,34 @@ The Apache Software License, Version 2.0 - net.java.dev.jna-jna-jpms-5.12.1.jar - net.java.dev.jna-jna-platform-jpms-5.12.1.jar * BookKeeper -- org.apache.bookkeeper-bookkeeper-common-4.16.4.jar -- org.apache.bookkeeper-bookkeeper-common-allocator-4.16.4.jar -- org.apache.bookkeeper-bookkeeper-proto-4.16.4.jar -- org.apache.bookkeeper-bookkeeper-server-4.16.4.jar -- org.apache.bookkeeper-bookkeeper-tools-framework-4.16.4.jar -- org.apache.bookkeeper-circe-checksum-4.16.4.jar -- org.apache.bookkeeper-cpu-affinity-4.16.4.jar -- org.apache.bookkeeper-statelib-4.16.4.jar -- org.apache.bookkeeper-stream-storage-api-4.16.4.jar -- org.apache.bookkeeper-stream-storage-common-4.16.4.jar -- org.apache.bookkeeper-stream-storage-java-client-4.16.4.jar -- org.apache.bookkeeper-stream-storage-java-client-base-4.16.4.jar -- org.apache.bookkeeper-stream-storage-proto-4.16.4.jar -- org.apache.bookkeeper-stream-storage-server-4.16.4.jar -- org.apache.bookkeeper-stream-storage-service-api-4.16.4.jar -- org.apache.bookkeeper-stream-storage-service-impl-4.16.4.jar -- org.apache.bookkeeper.http-http-server-4.16.4.jar -- org.apache.bookkeeper.http-vertx-http-server-4.16.4.jar -- org.apache.bookkeeper.stats-bookkeeper-stats-api-4.16.4.jar -- org.apache.bookkeeper.stats-prometheus-metrics-provider-4.16.4.jar -- org.apache.distributedlog-distributedlog-common-4.16.4.jar -- org.apache.distributedlog-distributedlog-core-4.16.4-tests.jar -- org.apache.distributedlog-distributedlog-core-4.16.4.jar -- org.apache.distributedlog-distributedlog-protocol-4.16.4.jar -- org.apache.bookkeeper.stats-codahale-metrics-provider-4.16.4.jar -- org.apache.bookkeeper-bookkeeper-slogger-api-4.16.4.jar -- org.apache.bookkeeper-bookkeeper-slogger-slf4j-4.16.4.jar -- org.apache.bookkeeper-native-io-4.16.4.jar +- org.apache.bookkeeper-bookkeeper-common-4.16.5.jar +- org.apache.bookkeeper-bookkeeper-common-allocator-4.16.5.jar +- org.apache.bookkeeper-bookkeeper-proto-4.16.5.jar +- org.apache.bookkeeper-bookkeeper-server-4.16.5.jar +- org.apache.bookkeeper-bookkeeper-tools-framework-4.16.5.jar +- org.apache.bookkeeper-circe-checksum-4.16.5.jar +- org.apache.bookkeeper-cpu-affinity-4.16.5.jar +- org.apache.bookkeeper-statelib-4.16.5.jar +- org.apache.bookkeeper-stream-storage-api-4.16.5.jar +- org.apache.bookkeeper-stream-storage-common-4.16.5.jar +- org.apache.bookkeeper-stream-storage-java-client-4.16.5.jar +- org.apache.bookkeeper-stream-storage-java-client-base-4.16.5.jar +- org.apache.bookkeeper-stream-storage-proto-4.16.5.jar +- org.apache.bookkeeper-stream-storage-server-4.16.5.jar +- org.apache.bookkeeper-stream-storage-service-api-4.16.5.jar +- org.apache.bookkeeper-stream-storage-service-impl-4.16.5.jar +- org.apache.bookkeeper.http-http-server-4.16.5.jar +- org.apache.bookkeeper.http-vertx-http-server-4.16.5.jar +- org.apache.bookkeeper.stats-bookkeeper-stats-api-4.16.5.jar +- org.apache.bookkeeper.stats-prometheus-metrics-provider-4.16.5.jar +- org.apache.distributedlog-distributedlog-common-4.16.5.jar +- org.apache.distributedlog-distributedlog-core-4.16.5-tests.jar +- org.apache.distributedlog-distributedlog-core-4.16.5.jar +- org.apache.distributedlog-distributedlog-protocol-4.16.5.jar +- org.apache.bookkeeper.stats-codahale-metrics-provider-4.16.5.jar +- org.apache.bookkeeper-bookkeeper-slogger-api-4.16.5.jar +- org.apache.bookkeeper-bookkeeper-slogger-slf4j-4.16.5.jar +- org.apache.bookkeeper-native-io-4.16.5.jar * Apache HTTP Client - org.apache.httpcomponents-httpclient-4.5.13.jar - org.apache.httpcomponents-httpcore-4.4.15.jar diff --git a/distribution/shell/src/assemble/LICENSE.bin.txt b/distribution/shell/src/assemble/LICENSE.bin.txt index
Re: [I] [Bug] Bookkeeper returns mac mismatch while reading [pulsar]
lhotari closed issue #21421: [Bug] Bookkeeper returns mac mismatch while reading URL: https://github.com/apache/pulsar/issues/21421 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar-site) branch dependabot/npm_and_yarn/tar-6.2.1 deleted (was 6544757a94e6)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/npm_and_yarn/tar-6.2.1 in repository https://gitbox.apache.org/repos/asf/pulsar-site.git was 6544757a94e6 build(deps): Bump tar from 6.2.0 to 6.2.1 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
Re: [PR] build(deps): Bump tar from 6.2.0 to 6.2.1 [pulsar-site]
visortelle merged PR #881: URL: https://github.com/apache/pulsar-site/pull/881 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar-site) branch main updated: build(deps): Bump tar from 6.2.0 to 6.2.1 (#881)
This is an automated email from the ASF dual-hosted git repository. visortelle pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/pulsar-site.git The following commit(s) were added to refs/heads/main by this push: new 965367edbd5f build(deps): Bump tar from 6.2.0 to 6.2.1 (#881) 965367edbd5f is described below commit 965367edbd5f52fdae1baddf7b6ea6c7ab6e69f0 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Fri Apr 12 10:12:24 2024 +0300 build(deps): Bump tar from 6.2.0 to 6.2.1 (#881) Bumps [tar](https://github.com/isaacs/node-tar) from 6.2.0 to 6.2.1. - [Release notes](https://github.com/isaacs/node-tar/releases) - [Changelog](https://github.com/isaacs/node-tar/blob/main/CHANGELOG.md) - [Commits](https://github.com/isaacs/node-tar/compare/v6.2.0...v6.2.1) --- updated-dependencies: - dependency-name: tar dependency-type: indirect ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- yarn.lock | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/yarn.lock b/yarn.lock index dd208b8ba586..d4c9059b4446 100644 --- a/yarn.lock +++ b/yarn.lock @@ -13867,8 +13867,8 @@ __metadata: linkType: hard "tar@npm:^6.1.0, tar@npm:^6.1.11, tar@npm:^6.1.2": - version: 6.2.0 - resolution: "tar@npm:6.2.0" + version: 6.2.1 + resolution: "tar@npm:6.2.1" dependencies: chownr: ^2.0.0 fs-minipass: ^2.0.0 @@ -13876,7 +13876,7 @@ __metadata: minizlib: ^2.1.1 mkdirp: ^1.0.3 yallist: ^4.0.0 - checksum: db4d9fe74a2082c3a5016630092c54c8375ff3b280186938cfd104f2e089c4fd9bad58688ef6be9cf186a889671bf355c7cda38f09bbf60604b281715ca57f5c + checksum: f1322768c9741a25356c11373bce918483f40fa9a25c69c59410c8a1247632487edef5fe76c5f12ac51a6356d2f1829e96d2bc34098668a2fc34d76050ac2b6c languageName: node linkType: hard
Re: [I] backlog quota exceed limit generates lot of logs within short period of time [pulsar]
nicoloboschi closed issue #22474: backlog quota exceed limit generates lot of logs within short period of time URL: https://github.com/apache/pulsar/issues/22474 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch branch-3.1 updated: [improve][broker] backlog quota exceed limit log replaced with `debug` (#22475)
This is an automated email from the ASF dual-hosted git repository.
nicoloboschi pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new 767f0b282a2 [improve][broker] backlog quota exceed limit log replaced
with `debug` (#22475)
767f0b282a2 is described below
commit 767f0b282a27581dfc16df9295ed01dd0df0a350
Author: Mukesh Kumar <65598381+mukesh...@users.noreply.github.com>
AuthorDate: Fri Apr 12 12:28:56 2024 +0530
[improve][broker] backlog quota exceed limit log replaced with `debug`
(#22475)
---
.../org/apache/pulsar/broker/service/persistent/PersistentTopic.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
index 5bceed099c1..ee367750ad6 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
@@ -2938,14 +2938,14 @@ public class PersistentTopic extends AbstractTopic
implements Topic, AddEntryCal
if ((retentionPolicy ==
BacklogQuota.RetentionPolicy.producer_request_hold
|| retentionPolicy ==
BacklogQuota.RetentionPolicy.producer_exception)) {
if (backlogQuotaType == BacklogQuotaType.destination_storage
&& isSizeBacklogExceeded()) {
-log.info("[{}] Size backlog quota exceeded. Cannot create
producer [{}]", this.getName(),
+log.debug("[{}] Size backlog quota exceeded. Cannot create
producer [{}]", this.getName(),
producerName);
return FutureUtil.failedFuture(new
TopicBacklogQuotaExceededException(retentionPolicy));
}
if (backlogQuotaType == BacklogQuotaType.message_age) {
return checkTimeBacklogExceeded().thenCompose(isExceeded
-> {
if (isExceeded) {
-log.info("[{}] Time backlog quota exceeded. Cannot
create producer [{}]", this.getName(),
+log.debug("[{}] Time backlog quota exceeded.
Cannot create producer [{}]", this.getName(),
producerName);
return FutureUtil.failedFuture(new
TopicBacklogQuotaExceededException(retentionPolicy));
} else {
Re: [PR] [improve][broker] backlog quota exceed limit log replaced with `debug` [pulsar]
nicoloboschi merged PR #22475: URL: https://github.com/apache/pulsar/pull/22475 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [D] pulsar broker load imbalance problem [pulsar]
GitHub user waney316 added a comment to the discussion: pulsar broker load imbalance problem > Any reason why these are disabled? I think [their defaults are to be > enabled](https://github.com/apache/pulsar/blob/cea1a9ba9b576bf43f0a45ff8d65369b0f2bbb36/conf/broker.conf#L1296C1-L1296C35)... > >  > > Is it possible those switches being off prevents the system from splitting > load across brokers? > > See here -> > https://pulsar.apache.org/docs/3.2.x/administration-load-balance/#split-namespace-bundles Because previously considered turning on loadBalancerAutoBundleSplitEnabled would cause client fluctuations，so the configuration items of 6 brokers are all true，If there is a problem with this configuration, why there is no big difference in the number of bundles on each broker? GitHub link: https://github.com/apache/pulsar/discussions/22483#discussioncomment-9091871 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
