date:20240508



nodece merged PR #22584:
URL: https://github.com/apache/pulsar/pull/22584


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [fix][sec] Upgrade Debezium oracle connector version to avoid CVE-2023-4586 [pulsar]



nikhilerigila09 commented on PR #22641:
URL: https://github.com/apache/pulsar/pull/22641#issuecomment-2101987908

   @lhotari 
   The CI check failed in `CI Flaky - System - Pulsar IO - Oracle`. I haven't 
tested it, but i see an issue open for flaky test - 
https://github.com/apache/pulsar/issues/13953


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(pulsar-client-go) branch master updated: [improve] Upgrade golang-jwt to v5 (#1214)

2024-05-08 Thread zixuan

This is an automated email from the ASF dual-hosted git repository.

zixuan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar-client-go.git


The following commit(s) were added to refs/heads/master by this push:
 new 007d14e5 [improve] Upgrade golang-jwt to v5 (#1214)
007d14e5 is described below

commit 007d14e50c886a0bbfc73d2a43d3cc932c9221d8
Author: Zixuan Liu 
AuthorDate: Thu May 9 13:46:42 2024 +0800

[improve] Upgrade golang-jwt to v5 (#1214)
---
 go.mod | 2 +-
 go.sum | 3 ++-
 oauth2/auth.go | 2 +-
 3 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/go.mod b/go.mod
index 52255512..b0f822f3 100644
--- a/go.mod
+++ b/go.mod
@@ -9,7 +9,6 @@ require (
github.com/bits-and-blooms/bitset v1.4.0
github.com/bmizerany/perks v0.0.0-20141205001514-d9a9656a3a4b
github.com/davecgh/go-spew v1.1.1
-   github.com/golang-jwt/jwt v3.2.1+incompatible
github.com/google/uuid v1.1.2
github.com/klauspost/compress v1.14.4
github.com/linkedin/goavro/v2 v2.9.8
@@ -31,6 +30,7 @@ require (
 )
 
 require (
+   github.com/golang-jwt/jwt/v5 v5.2.1
github.com/golang/protobuf v1.5.2
github.com/hashicorp/go-multierror v1.1.1
 )
diff --git a/go.sum b/go.sum
index f48b384a..64f5faab 100644
--- a/go.sum
+++ b/go.sum
@@ -98,8 +98,9 @@ github.com/go-task/slim-sprig 
v0.0.0-20210107165309-348f09dbbbc0/go.mod h1:fyg78
 github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2 
h1:ZpnhV/YsD2/4cESfV5+Hoeu/iUR3ruzNvZ+yQfO03a0=
 github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2/go.mod 
h1:bBOAhwG1umN6/6ZUMtDFBMQR8jRg9O75tm9K00oMsK4=
 github.com/gogo/protobuf v1.1.1/go.mod 
h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
-github.com/golang-jwt/jwt v3.2.1+incompatible 
h1:73Z+4BJcrTC+KczS6WvTPvRGOp1WmfEP4Q1lOd9Z/+c=
 github.com/golang-jwt/jwt v3.2.1+incompatible/go.mod 
h1:8pz2t5EyA70fFQQSrl6XZXzqecmYZeUEB8OUGHkxJ+I=
+github.com/golang-jwt/jwt/v5 v5.2.1 
h1:OuVbFODueb089Lh128TAcimifWaLhJwVflnrgM17wHk=
+github.com/golang-jwt/jwt/v5 v5.2.1/go.mod 
h1:pqrtFR0X4osieyHYxtmOUWsAWrfe1Q5UVIyoH402zdk=
 github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod 
h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
 github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod 
h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
 github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod 
h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
diff --git a/oauth2/auth.go b/oauth2/auth.go
index d44bd350..9f4293b5 100644
--- a/oauth2/auth.go
+++ b/oauth2/auth.go
@@ -22,7 +22,7 @@ import (
"time"
 
"github.com/apache/pulsar-client-go/oauth2/clock"
-   "github.com/golang-jwt/jwt"
+   "github.com/golang-jwt/jwt/v5"
"golang.org/x/oauth2"
 )

Re: [PR] [improve] Upgrade golang-jwt to v5 [pulsar-client-go]



nodece merged PR #1214:
URL: https://github.com/apache/pulsar-client-go/pull/1214


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] [Bug] parseMessageMetadata error when broker entry metadata enable with high loading [pulsar]



semistone commented on issue #22601:
URL: https://github.com/apache/pulsar/issues/22601#issuecomment-2101968443

   Hi @lhotari 
I update perf tool in 
   https://github.com/semistone/pulsar/tree/debug_ssues_22601
   
   it only include one commit  which modify PerformanceProducer.java to include 
   big payload ( -bp 5 means 5 percent big payload)
   and BatcherBuilder.KEY_BASED (-kb)
   
   
   my consumer command is 
   ```
   bin/pulsar-perf  consume persistent://my-tenant/my-namespace/my-topic-1   
--auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationTls 
--auth-params 
'{"tlsCertFile":"conf/superuser.cer","tlsKeyFile":"conf/superuser.key.pem"}' -n 
10 -sp Latest -ss angus_test --batch-index-ack   -st Key_Shared 
   ```
   
   and producer command is 
   ```
   bin/pulsar-perf produce persistent://my-tenant/my-namespace/my-topic-1 -r 
6000 -kb -s 2000 -bp 5  -bm 1000  -b 1 -mk random  --auth-plugin 
org.apache.pulsar.client.impl.auth.AuthenticationTls --auth-params 
'{"tlsCertFile":"conf/superuser.cer","tlsKeyFile":"conf/superuser.key.pem"}' 
   ```
   
   that error happen when
   
   Batch builder is KEY_BASE
   with random event key
   and few big payload (in my environment 3% could reproduce 10% will crash 
producer)
   
   in my test
   I use normal payload 2K bytes , big payload 20K bytes
   if I removed any above conditions, that error will either reduced or 
disappear.
   
   when it happen it will have WARN message in pulsar-broker.log
   ```
   2024-05-09T01:12:35,246+ [pulsar-io-3-31] WARN  
org.apache.pulsar.broker.service.ServerCnx - [/100.96.184.253:39710] Got 
exception java.lang.IllegalArgumentException: Invalid unknonwn tag type: 6
   or 
   2024-05-09T01:12:35,260+ [broker-topic-workers-OrderedExecutor-15-0] 
ERROR org.apache.pulsar.common.protocol.Commands - 
[persistent://budas/budas-preprod-internal/bud_stream_input-partition-1] 
[angus_test] Failed to peek sticky key from the message metadata
   java.lang.IllegalArgumentException: Invalid unknonwn tag type: 4
   ```
   unfortunately I can't preproduce in docker, I guess docker standalone is 
different from my pulsar cluster.
   my pulsar cluster is 
   almost default config but with TLS auth in broker/bookkeeper/zookeeper.
   
   Please help to check it, if have any problem to reproduce this issue in your 
environment, 
   then I will try to simplify my pulsar cluster to reproduce it.

   Thanks
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [fix][broker] Fix geo-replication admin client url [pulsar]



codecov-commenter commented on PR #22584:
URL: https://github.com/apache/pulsar/pull/22584#issuecomment-2101949371

   ## 
[Codecov](https://app.codecov.io/gh/apache/pulsar/pull/22584?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
 Report
   Attention: Patch coverage is `33.3%` with `2 lines` in your changes are 
missing coverage. Please review.
   > Project coverage is 73.13%. Comparing base 
[(`bbc6224`)](https://app.codecov.io/gh/apache/pulsar/commit/bbc62245c5ddba1de4b1e7cee4ab49334bc36277?dropdown=coverage=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
 to head 
[(`6b0dc19`)](https://app.codecov.io/gh/apache/pulsar/pull/22584?dropdown=coverage=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache).
   > Report is 246 commits behind head on master.
   
   Additional details and impacted files
   
   
   [![Impacted file tree 
graph](https://app.codecov.io/gh/apache/pulsar/pull/22584/graphs/tree.svg?width=650=150=pr=acYqCpsK9J_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)](https://app.codecov.io/gh/apache/pulsar/pull/22584?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
   
   ```diff
   @@ Coverage Diff  @@
   ## master   #22584  +/-   ##
   
   - Coverage 73.57%   73.13%   -0.45% 
   - Complexity3262432772 +148 
   
 Files  1877 1887  +10 
 Lines139502   141073+1571 
 Branches  1529915484 +185 
   
   + Hits 102638   103168 +530 
   - Misses2890829949+1041 
 Partials   7956 7956  
   ```
   
   | 
[Flag](https://app.codecov.io/gh/apache/pulsar/pull/22584/flags?src=pr=flags_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
 | Coverage Δ | |
   |---|---|---|
   | 
[inttests](https://app.codecov.io/gh/apache/pulsar/pull/22584/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
 | `27.32% <0.00%> (+2.73%)` | :arrow_up: |
   | 
[systests](https://app.codecov.io/gh/apache/pulsar/pull/22584/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
 | `24.58% <0.00%> (+0.25%)` | :arrow_up: |
   | 
[unittests](https://app.codecov.io/gh/apache/pulsar/pull/22584/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
 | `72.15% <33.33%> (-0.70%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click 
here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#carryforward-flags-in-the-pull-request-comment)
 to find out more.
   
   | 
[Files](https://app.codecov.io/gh/apache/pulsar/pull/22584?dropdown=coverage=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
 | Coverage Δ | |
   |---|---|---|
   | 
[...rg/apache/pulsar/broker/service/BrokerService.java](https://app.codecov.io/gh/apache/pulsar/pull/22584?src=pr=tree=pulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fservice%2FBrokerService.java_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zZXJ2aWNlL0Jyb2tlclNlcnZpY2UuamF2YQ==)
 | `81.45% <33.33%> (+0.67%)` | :arrow_up: |
   
   ... and [340 files with indirect coverage 
changes](https://app.codecov.io/gh/apache/pulsar/pull/22584/indirect-changes?src=pr=tree-more_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] [Bug] parseMessageMetadata error when broker entry metadata enable with high loading [pulsar]



semistone commented on issue #22601:
URL: https://github.com/apache/pulsar/issues/22601#issuecomment-2101947995

   Hi @lhotari 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [fix][broker] fix replicated subscriptions for transactional messages [pulsar]



thetumbled commented on PR #22452:
URL: https://github.com/apache/pulsar/pull/22452#issuecomment-2101877822

   /pulsarbot rerun-failure-checks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [fix][broker] Fix geo-replication admin client url [pulsar]



nodece commented on PR #22584:
URL: https://github.com/apache/pulsar/pull/22584#issuecomment-2101778525

   CI failed: 
https://github.com/apache/pulsar/actions/runs/9001640819/job/24737668955?pr=22584#step:11:1612


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(pulsar) branch master updated: [fix][broker] usedLocallySinceLastReport should always be reset (#22672)

2024-05-08 Thread zixuan

This is an automated email from the ASF dual-hosted git repository.

zixuan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 8f015d89e5d [fix][broker] usedLocallySinceLastReport should always be 
reset (#22672)
8f015d89e5d is described below

commit 8f015d89e5d246325ae5cada02c4af3017a97ed9
Author: Zixuan Liu 
AuthorDate: Thu May 9 09:42:17 2024 +0800

[fix][broker] usedLocallySinceLastReport should always be reset (#22672)

Signed-off-by: Zixuan Liu 
---
 .../pulsar/broker/resourcegroup/ResourceGroup.java |  3 +-
 .../ResourceGroupReportLocalUsageTest.java | 50 ++
 2 files changed, 34 insertions(+), 19 deletions(-)

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/resourcegroup/ResourceGroup.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/resourcegroup/ResourceGroup.java
index f8ec52bfe3c..541a645f18b 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/resourcegroup/ResourceGroup.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/resourcegroup/ResourceGroup.java
@@ -458,14 +458,13 @@ public class ResourceGroup {
 
 bytesUsed = monEntity.usedLocallySinceLastReport.bytes;
 messagesUsed = monEntity.usedLocallySinceLastReport.messages;
-
+monEntity.usedLocallySinceLastReport.bytes = 
monEntity.usedLocallySinceLastReport.messages = 0;
 if (sendReport) {
 p.setBytesPerPeriod(bytesUsed);
 p.setMessagesPerPeriod(messagesUsed);
 monEntity.lastReportedValues.bytes = bytesUsed;
 monEntity.lastReportedValues.messages = messagesUsed;
 monEntity.numSuppressedUsageReports = 0;
-monEntity.usedLocallySinceLastReport.bytes = 
monEntity.usedLocallySinceLastReport.messages = 0;
 monEntity.totalUsedLocally.bytes += bytesUsed;
 monEntity.totalUsedLocally.messages += messagesUsed;
 monEntity.lastResourceUsageFillTimeMSecsSinceEpoch = 
System.currentTimeMillis();
diff --git 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/resourcegroup/ResourceGroupReportLocalUsageTest.java
 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/resourcegroup/ResourceGroupReportLocalUsageTest.java
index 658b7c94165..139d19886c7 100644
--- 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/resourcegroup/ResourceGroupReportLocalUsageTest.java
+++ 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/resourcegroup/ResourceGroupReportLocalUsageTest.java
@@ -72,34 +72,50 @@ public class ResourceGroupReportLocalUsageTest extends 
MockedPulsarServiceBaseTe
 rgConfig.setPublishRateInMsgs(2000);
 service.resourceGroupCreate(rgName, rgConfig);
 
-org.apache.pulsar.broker.resourcegroup.ResourceGroup resourceGroup = 
service.resourceGroupGet(rgName);
 BytesAndMessagesCount bytesAndMessagesCount = new 
BytesAndMessagesCount();
 bytesAndMessagesCount.bytes = 20;
 bytesAndMessagesCount.messages = 10;
-
resourceGroup.incrementLocalUsageStats(ResourceGroupMonitoringClass.Publish, 
bytesAndMessagesCount);
+
+org.apache.pulsar.broker.resourcegroup.ResourceGroup resourceGroup = 
service.resourceGroupGet(rgName);
+for (ResourceGroupMonitoringClass value : 
ResourceGroupMonitoringClass.values()) {
+resourceGroup.incrementLocalUsageStats(value, 
bytesAndMessagesCount);
+}
+
+// Case1: Suppress report ResourceUsage.
+needReport.set(false);
 ResourceUsage resourceUsage = new ResourceUsage();
 resourceGroup.rgFillResourceUsage(resourceUsage);
 assertFalse(resourceUsage.hasDispatch());
 assertFalse(resourceUsage.hasPublish());
+for (ResourceGroupMonitoringClass value : 
ResourceGroupMonitoringClass.values()) {
+PerMonitoringClassFields monitoredEntity =
+resourceGroup.getMonitoredEntity(value);
+assertEquals(monitoredEntity.usedLocallySinceLastReport.messages, 
0);
+assertEquals(monitoredEntity.usedLocallySinceLastReport.bytes, 0);
+assertEquals(monitoredEntity.totalUsedLocally.messages, 0);
+assertEquals(monitoredEntity.totalUsedLocally.bytes, 0);
+assertEquals(monitoredEntity.lastReportedValues.messages, 0);
+assertEquals(monitoredEntity.lastReportedValues.bytes, 0);
+}
 
-PerMonitoringClassFields publishMonitoredEntity =
-
resourceGroup.getMonitoredEntity(ResourceGroupMonitoringClass.Publish);
-
assertEquals(publishMonitoredEntity.usedLocallySinceLastReport.messages, 
bytesAndMessagesCount.messages);
-assertEquals(publishMonitoredEntity.usedLocallySinceLastReport.bytes, 
bytesAndMessagesCount.bytes);
-

Re: [PR] [fix][broker] usedLocallySinceLastReport should always be reset [pulsar]



nodece merged PR #22672:
URL: https://github.com/apache/pulsar/pull/22672


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] [feature request] Support publish messages to partitioned topics [pulsar-client-node]



merlimat commented on issue #383:
URL: 
https://github.com/apache/pulsar-client-node/issues/383#issuecomment-2101775376

   Client is based on C++ lib which already supports publishing to partitioned 
topics transparently.
   
   The routing mode can be selected with the `messageRoutingMode` property when 
creating the producer.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(pulsar-site) branch main updated: Docs sync done from apache/pulsar (#88feb87)

2024-05-08 Thread urfree

This is an automated email from the ASF dual-hosted git repository.

urfree pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/pulsar-site.git


The following commit(s) were added to refs/heads/main by this push:
 new edfda8ee2cbb Docs sync done from apache/pulsar (#88feb87)
edfda8ee2cbb is described below

commit edfda8ee2cbb2e7626a41f6101055ece313e4922
Author: github-actions[bot] 
<41898282+github-actions[bot]@users.noreply.github.com>
AuthorDate: Thu May 9 01:34:12 2024 +

Docs sync done from apache/pulsar (#88feb87)
---
 .../reference/next/config/reference-configuration-broker.md   | 11 +++
 .../next/config/reference-configuration-standalone.md | 11 +++
 .../next/config/reference-configuration-websocket.md  | 11 +++
 3 files changed, 33 insertions(+)

diff --git a/static/reference/next/config/reference-configuration-broker.md 
b/static/reference/next/config/reference-configuration-broker.md
index 2545e65d0b81..4691c9620628 100644
--- a/static/reference/next/config/reference-configuration-broker.md
+++ b/static/reference/next/config/reference-configuration-broker.md
@@ -5445,6 +5445,17 @@ Interval of time to sending the ping to keep alive in 
WebSocket proxy. This valu
 
 **Category**: WebSocket
 
+### webSocketPulsarClientMemoryLimitInMB
+Memory limit in MBs for direct memory in Pulsar Client used in WebSocket proxy
+
+**Type**: `int`
+
+**Default**: `0`
+
+**Dynamic**: `false`
+
+**Category**: WebSocket
+
 ### webSocketServiceEnabled
 Enable the WebSocket API service in broker
 
diff --git a/static/reference/next/config/reference-configuration-standalone.md 
b/static/reference/next/config/reference-configuration-standalone.md
index 2545e65d0b81..4691c9620628 100644
--- a/static/reference/next/config/reference-configuration-standalone.md
+++ b/static/reference/next/config/reference-configuration-standalone.md
@@ -5445,6 +5445,17 @@ Interval of time to sending the ping to keep alive in 
WebSocket proxy. This valu
 
 **Category**: WebSocket
 
+### webSocketPulsarClientMemoryLimitInMB
+Memory limit in MBs for direct memory in Pulsar Client used in WebSocket proxy
+
+**Type**: `int`
+
+**Default**: `0`
+
+**Dynamic**: `false`
+
+**Category**: WebSocket
+
 ### webSocketServiceEnabled
 Enable the WebSocket API service in broker
 
diff --git a/static/reference/next/config/reference-configuration-websocket.md 
b/static/reference/next/config/reference-configuration-websocket.md
index fec6018fe780..3881a32cba4f 100644
--- a/static/reference/next/config/reference-configuration-websocket.md
+++ b/static/reference/next/config/reference-configuration-websocket.md
@@ -613,6 +613,17 @@ Interval of time to sending the ping to keep alive. This 
value greater than 0 me
 
 **Category**: 
 
+### webSocketPulsarClientMemoryLimitInMB
+Memory limit in MBs for direct memory in Pulsar Client used in WebSocket proxy
+
+**Type**: `int`
+
+**Default**: `0`
+
+**Dynamic**: `false`
+
+**Category**: 
+
 ### webSocketSessionIdleTimeoutMillis
 Timeout of idling WebSocket session (in milliseconds)

[I] [feature request] Support publish messages to partitioned topics [pulsar-client-node]



yash2 opened a new issue, #383:
URL: https://github.com/apache/pulsar-client-node/issues/383

   Could we provide support for node clients to publish messages partition 
topics,
   - So publishing messages to partitioned topics, can specify a routing mode. 
If we not specify any routing mode then round-robin routing mode should be used.
   
   Reference docs supporting the functionality for other client libraries: 
https://pulsar.apache.org/docs/next/client-libraries-producers/#publish-messages-to-partitioned-topics


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [fix][broker] Fix cursor should use latest ledger config [pulsar]



Technoboy- commented on code in PR #22644:
URL: https://github.com/apache/pulsar/pull/22644#discussion_r1594850056


##
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java:
##
@@ -299,31 +298,30 @@ public interface VoidCallback {
 void operationFailed(ManagedLedgerException exception);
 }
 
-ManagedCursorImpl(BookKeeper bookkeeper, ManagedLedgerConfig config, 
ManagedLedgerImpl ledger, String cursorName) {
+ManagedCursorImpl(BookKeeper bookkeeper, ManagedLedgerImpl ledger, String 
cursorName) {
 this.bookkeeper = bookkeeper;
 this.cursorProperties = Collections.emptyMap();
-this.config = config;

Review Comment:
   this.config = ledger.getConfig();



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [fix][broker] Fix cursor should use latest ledger config [pulsar]



Technoboy- commented on code in PR #22644:
URL: https://github.com/apache/pulsar/pull/22644#discussion_r1594850056


##
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java:
##
@@ -299,31 +298,30 @@ public interface VoidCallback {
 void operationFailed(ManagedLedgerException exception);
 }
 
-ManagedCursorImpl(BookKeeper bookkeeper, ManagedLedgerConfig config, 
ManagedLedgerImpl ledger, String cursorName) {
+ManagedCursorImpl(BookKeeper bookkeeper, ManagedLedgerImpl ledger, String 
cursorName) {
 this.bookkeeper = bookkeeper;
 this.cursorProperties = Collections.emptyMap();
-this.config = config;

Review Comment:
   this.config = ledger.getConfig();



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [fix][ml] Remove duplicated field initialization of ML [pulsar]



Technoboy- merged PR #22676:
URL: https://github.com/apache/pulsar/pull/22676


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(pulsar) branch master updated: [fix][ml] Remove duplicated field initialization of ML (#22676)

This is an automated email from the ASF dual-hosted git repository.

technoboy pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 88feb874bb3 [fix][ml] Remove duplicated field initialization of ML 
(#22676)
88feb874bb3 is described below

commit 88feb874bb3ad58a74b3d40d931b2aa7380dc7e1
Author: 道君 
AuthorDate: Thu May 9 08:53:59 2024 +0800

[fix][ml] Remove duplicated field initialization of ML (#22676)
---
 .../java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java | 3 ---
 1 file changed, 3 deletions(-)

diff --git 
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
 
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
index e5e163127f7..b12346cadc9 100644
--- 
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
+++ 
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
@@ -365,9 +365,6 @@ public class ManagedLedgerImpl implements ManagedLedger, 
CreateCallback {
 this.mlOwnershipChecker = mlOwnershipChecker;
 this.propertiesMap = new ConcurrentHashMap<>();
 this.inactiveLedgerRollOverTimeMs = 
config.getInactiveLedgerRollOverTimeMs();
-if (config.getManagedLedgerInterceptor() != null) {
-this.managedLedgerInterceptor = 
config.getManagedLedgerInterceptor();
-}
 this.minBacklogCursorsForCaching = 
config.getMinimumBacklogCursorsForCaching();
 this.minBacklogEntriesForCaching = 
config.getMinimumBacklogEntriesForCaching();
 this.maxBacklogBetweenCursorsForCaching = 
config.getMaxBacklogBetweenCursorsForCaching();

Re: [PR] [feat][sql] Support querying compacted data in Pulsar SQL [pulsar]



tisonkun commented on PR #19515:
URL: https://github.com/apache/pulsar/pull/19515#issuecomment-2101711636

   Closed. We no longer bundle pulsar-sql.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [feat][sql] Support querying compacted data in Pulsar SQL [pulsar]



tisonkun closed pull request #19515: [feat][sql] Support querying compacted 
data in Pulsar SQL
URL: https://github.com/apache/pulsar/pull/19515


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [improve][cli]Pulsar Dynamic Configuration Feature for Broker-level Configuration [pulsar]



tisonkun commented on PR #20052:
URL: https://github.com/apache/pulsar/pull/20052#issuecomment-2101711155

   Closed as stale and conflict.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [improve][cli]Pulsar Dynamic Configuration Feature for Broker-level Configuration [pulsar]



tisonkun closed pull request #20052: [improve][cli]Pulsar Dynamic Configuration 
Feature for Broker-level Configuration
URL: https://github.com/apache/pulsar/pull/20052


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] [Bug] Elasticsearch Java version has a vulnerability CVE-2023-46673, CVE-2023-46674, CVE-2023-31419, CVE-2023-31417, CVE-2023-31418 [pulsar]



hpvd commented on issue #21782:
URL: https://github.com/apache/pulsar/issues/21782#issuecomment-2101572900

   see also ongoing discussion 
https://github.com/apache/pulsar/discussions/19093#discussioncomment-9360551
   and upvote it if you also think this is important.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] [Bug] Alluxio version has a vulnerability CVE-2023-38889 [pulsar]



hpvd commented on issue #21780:
URL: https://github.com/apache/pulsar/issues/21780#issuecomment-2101570459

   see also ongoing discussion 
https://github.com/apache/pulsar/discussions/19093#discussioncomment-9360551
   and upvote it if you also think this is important.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [D] Automated security and update routine before every release [pulsar]



GitHub user hpvd edited a comment on the discussion: Automated security and 
update routine before every release

current OWASP dependency checks on merge always fails with many 
errors/vulnerabilities with a high CVSS score greater or equal than 7,
(several scores are even greater than 9,5)
but do NOT block
please see example attached
-> so we may want to think of simply removing it to save some load/time?

only as typical example:

![2024-05-08_23h42_38](https://github.com/apache/pulsar/assets/5681880/6e0136d7-ebfd-40fa-bf33-43b57a0a062c)


![2024-05-08_23h33_43](https://github.com/apache/pulsar/assets/5681880/b6275511-28a5-4bda-8ada-1b94d9902732)
https://github.com/apache/pulsar/actions/runs/8980506133/job/24665993927






GitHub link: 
https://github.com/apache/pulsar/discussions/19093#discussioncomment-9360551


This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org

Re: [D] Automated security and update routine before every release [pulsar]



GitHub user hpvd edited a comment on the discussion: Automated security and 
update routine before every release

current OWASP dependency checks on merge always fails with many 
errors/vulnerabilities with a high CVSS score greater or equal than 7,
(several scores are even greater than 9,5)
but do NOT block
please see example attached
-> so we may want to think of simply removing it to save some load/time?

only as example

![2024-05-08_23h42_38](https://github.com/apache/pulsar/assets/5681880/6e0136d7-ebfd-40fa-bf33-43b57a0a062c)


![2024-05-08_23h33_43](https://github.com/apache/pulsar/assets/5681880/b6275511-28a5-4bda-8ada-1b94d9902732)
https://github.com/apache/pulsar/actions/runs/8980506133/job/24665993927






GitHub link: 
https://github.com/apache/pulsar/discussions/19093#discussioncomment-9360551


This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org

Re: [D] Automated security and update routine before every release [pulsar]



GitHub user hpvd edited a comment on the discussion: Automated security and 
update routine before every release

current OWASP dependency checks on merge always fails with many 
errors/vulnerabilities with a high CVSS score greater or equal than 7,
(several scores are even greater than 9,5)
but do NOT block
please see example attached
-> so we may think of simply removing it to save some load/time?

only as example

![2024-05-08_23h42_38](https://github.com/apache/pulsar/assets/5681880/6e0136d7-ebfd-40fa-bf33-43b57a0a062c)


![2024-05-08_23h33_43](https://github.com/apache/pulsar/assets/5681880/b6275511-28a5-4bda-8ada-1b94d9902732)
https://github.com/apache/pulsar/actions/runs/8980506133/job/24665993927






GitHub link: 
https://github.com/apache/pulsar/discussions/19093#discussioncomment-9360551


This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org

Re: [D] Automated security and update routine before every release [pulsar]



GitHub user hpvd edited a comment on the discussion: Automated security and 
update routine before every release

current OWASP dependency checks on merge always fails with many 
errors/vulnerabilities with a high CVSS score greater or equal than 7,
but do NOT block
please see example attached
-> so we may think of simply removing it to save some load/time?

only as example

![2024-05-08_23h42_38](https://github.com/apache/pulsar/assets/5681880/6e0136d7-ebfd-40fa-bf33-43b57a0a062c)


![2024-05-08_23h33_43](https://github.com/apache/pulsar/assets/5681880/b6275511-28a5-4bda-8ada-1b94d9902732)
https://github.com/apache/pulsar/actions/runs/8980506133/job/24665993927






GitHub link: 
https://github.com/apache/pulsar/discussions/19093#discussioncomment-9360551


This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org

Re: [D] Automated security and update routine before every release [pulsar]



GitHub user hpvd edited a comment on the discussion: Automated security and 
update routine before every release

current owasp dependency checks on merge always fails with many 
errors/vulnerabilities with a high CVSS score greater or equal than 7,
but do NOT block
please see example attached
-> so we may think of simply removing it to save some load/time?

only as example

![2024-05-08_23h42_38](https://github.com/apache/pulsar/assets/5681880/6e0136d7-ebfd-40fa-bf33-43b57a0a062c)


![2024-05-08_23h33_43](https://github.com/apache/pulsar/assets/5681880/b6275511-28a5-4bda-8ada-1b94d9902732)
https://github.com/apache/pulsar/actions/runs/8980506133/job/24665993927






GitHub link: 
https://github.com/apache/pulsar/discussions/19093#discussioncomment-9360551


This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org

Re: [D] Automated security and update routine before every release [pulsar]



GitHub user hpvd added a comment to the discussion: Automated security and 
update routine before every release

current owasp dependency checks on merge always fails with many 
errors/vulnerabilities with a high CVSS score greater or equal than 7,
but do NOT block
please see example attached
-> so we may think of simply removing it to save some load/time?

only as example
![2024-05-08_23h33_43](https://github.com/apache/pulsar/assets/5681880/b6275511-28a5-4bda-8ada-1b94d9902732)
https://github.com/apache/pulsar/actions/runs/8980506133/job/24665993927




GitHub link: 
https://github.com/apache/pulsar/discussions/19093#discussioncomment-9360551


This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org

(pulsar) branch master updated: [improve][broker] Remove unused method CompactionRecord.reset (#22670)

2024-05-08 Thread mmerli

This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new ad75e3f0921 [improve][broker] Remove unused method 
CompactionRecord.reset (#22670)
ad75e3f0921 is described below

commit ad75e3f0921bb735766d5e699baea0fc39ac4d41
Author: Dragos Misca 
AuthorDate: Wed May 8 13:54:16 2024 -0700

[improve][broker] Remove unused method CompactionRecord.reset (#22670)
---
 .../main/java/org/apache/pulsar/compaction/CompactionRecord.java  | 8 
 .../java/org/apache/pulsar/compaction/CompactorMXBeanImpl.java| 4 
 .../org/apache/pulsar/compaction/CompactorMXBeanImplTest.java | 5 -
 3 files changed, 17 deletions(-)

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactionRecord.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactionRecord.java
index 1d2af6638c3..cea005d51b8 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactionRecord.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactionRecord.java
@@ -51,14 +51,6 @@ public class CompactionRecord {
 public final Rate writeRate = new Rate();
 public final Rate readRate = new Rate();
 
-public void reset() {
-compactionRemovedEventCount.reset();
-compactionSucceedCount.reset();
-compactionFailedCount.reset();
-compactionDurationTimeInMills.reset();
-writeLatencyStats.reset();
-}
-
 public void addCompactionRemovedEvent() {
 lastCompactionRemovedEventCountOp.increment();
 compactionRemovedEventCount.increment();
diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactorMXBeanImpl.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactorMXBeanImpl.java
index 64b91d17d25..8a9d266b56e 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactorMXBeanImpl.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactorMXBeanImpl.java
@@ -53,10 +53,6 @@ public class CompactorMXBeanImpl implements CompactorMXBean {
 return compactionRecordOps.keySet();
 }
 
-public void reset() {
-compactionRecordOps.values().forEach(CompactionRecord::reset);
-}
-
 public void addCompactionReadOp(String topic, long readableBytes) {
 compactionRecordOps.computeIfAbsent(topic, k -> new 
CompactionRecord()).addCompactionReadOp(readableBytes);
 }
diff --git 
a/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactorMXBeanImplTest.java
 
b/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactorMXBeanImplTest.java
index bbde59d7da8..73e7430bd2d 100644
--- 
a/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactorMXBeanImplTest.java
+++ 
b/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactorMXBeanImplTest.java
@@ -59,11 +59,6 @@ public class CompactorMXBeanImplTest {
 assertTrue(compaction.getCompactionWriteThroughput() > 0L);
 mxBean.addCompactionLatencyOp(topic, 10, TimeUnit.NANOSECONDS);
 assertTrue(compaction.getCompactionLatencyBuckets()[0] > 0L);
-mxBean.reset();
-assertEquals(compaction.getCompactionRemovedEventCount(), 0, 0);
-assertEquals(compaction.getCompactionSucceedCount(), 0, 0);
-assertEquals(compaction.getCompactionFailedCount(), 0, 0);
-assertEquals(compaction.getCompactionDurationTimeInMills(), 0, 0);
 }
 
 }

Re: [PR] [improve][broker] Remove unused method CompactionRecord.reset [pulsar]



merlimat merged PR #22670:
URL: https://github.com/apache/pulsar/pull/22670


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] [Bug] consumers stops receiving new messages [pulsar]



lhotari commented on issue #22657:
URL: https://github.com/apache/pulsar/issues/22657#issuecomment-2101348655

   > * pulsar_subscription_blocked_on_unacked_messages metric shows 0
   
   sorry, noticed this now


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] [Bug] consumers stops receiving new messages [pulsar]



lhotari commented on issue #22657:
URL: https://github.com/apache/pulsar/issues/22657#issuecomment-2101346975

   @180254 in your case, can you detect the issue from topic stats? for 
example, does it tell `"blockedSubscriptionOnUnackedMsgs": true`?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [D] intermittent Messages loss while consuming the messages from consumer [pulsar]



GitHub user ragaur-tibco edited a comment on the discussion: intermittent 
Messages loss while consuming the messages from consumer

@lhotari @wallacepeng 

I just saw one defect on the message loss link: 
https://github.com/apache/pulsar/issues/22657

in my case I have used the above mentioned points(in my previous comment) after 
running the consumer to consume the messages saw that there are some loss of 
messages happening 

Like messages sent 1 to 10k
While running consumer duplication happen for ID's: 557, 880, 960, 990 
Message loss Id: 556 

NOTE: producer is closed after sending 10k messages.  and in standalone.conf 
file brokerDeduplicationEnabled: "false" 


GitHub link: 
https://github.com/apache/pulsar/discussions/22681#discussioncomment-9359474


This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org

Re: [D] intermittent Messages loss while consuming the messages from consumer [pulsar]



GitHub user ragaur-tibco edited a comment on the discussion: intermittent 
Messages loss while consuming the messages from consumer

@lhotari 

I just saw one defect on the message loss link: 
https://github.com/apache/pulsar/issues/22657

in my case I have used the above mentioned points(in my previous comment) after 
running the consumer to consume the messages saw that there are some loss of 
messages happening 

Like messages sent 1 to 10k
While running consumer duplication happen for ID's: 557, 880, 960, 990 
Message loss Id: 556 

NOTE: producer is closed after sending 10k messages.  and in standalone.conf 
file brokerDeduplicationEnabled: "false" 


GitHub link: 
https://github.com/apache/pulsar/discussions/22681#discussioncomment-9359474


This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org

Re: [D] intermittent Messages loss while consuming the messages from consumer [pulsar]



GitHub user ragaur-tibco edited a comment on the discussion: intermittent 
Messages loss while consuming the messages from consumer

@lhotari 

I just saw one defect on the message loss link: 
https://github.com/apache/pulsar/issues/22657

in my case I have used the above mentioned points(in my previous comment) after 
running the consumer to consume the messages saw that there are some loss of 
messages happening 

Like messages sent 1 to 10k
While running consumer duplication happen for ID's: 557, 880, 960, 990 
Message loss Id: 556 

NOTE: producer is closed after sending 10k messages. 


GitHub link: 
https://github.com/apache/pulsar/discussions/22681#discussioncomment-9359474


This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org

Re: [D] intermittent Messages loss while consuming the messages from consumer [pulsar]



GitHub user ragaur-tibco added a comment to the discussion: intermittent 
Messages loss while consuming the messages from consumer

@lhotari 

I just saw one defect on the message loss link: 
https://github.com/apache/pulsar/issues/22657

in my case I have used the above mentioned points(in my previous comment) after 
running the consumer to consume the messages saw that there are some loss of 
messages happening

NOTE: producer is closed after sending 10k messages. 

GitHub link: 
https://github.com/apache/pulsar/discussions/22681#discussioncomment-9359474


This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org

Re: [D] Not able to configure geo-replication with SSL [pulsar]



GitHub user visortelle added a comment to the discussion: Not able to configure 
geo-replication with SSL

@ragaur-tibco hi 
It would be **help a lot** if you could provide some simple way to reproduce 
exactly what you're doing.
Maybe in the form of a docker-compose.yml file and a shell script with all the 
certs generation and cluster registration steps.
Or Helm deployment that I could install to Minikube or k3s + scripts.

GitHub link: 
https://github.com/apache/pulsar/discussions/22605#discussioncomment-9359459


This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org

Re: [D] Not able to configure geo-replication with SSL [pulsar]



GitHub user ragaur-tibco added a comment to the discussion: Not able to 
configure geo-replication with SSL

@visortelle  any update or the cause of the this issue, I am using the same 
configuration for SSL but after creating cluster facing this issue in cluster B 
but on cluster A server was UP 

GitHub link: 
https://github.com/apache/pulsar/discussions/22605#discussioncomment-9359375


This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org

Re: [D] msgOutCounter parameter showing some values even I send messages from producer [pulsar]



GitHub user ragaur-tibco edited a discussion: msgOutCounter parameter showing 
some values even I send messages from producer

Hi @lhotari  @visortelle 

I have a query 

I tried to send messages 1 from the producer and when I see the topic stats 
it shows that msgInCounter -> 1 but in msgOutCounter -> 2000 why is this 
the case?

Please correct me if I am wrong
msgInCounter->  shows the number of messages present in a topic and send by the 
producer 
msgOutCounter -> shows the number of messages consumed by consumer 


So without running the consumer code is there any possibility that the values 
of msgOutCounter parameter increased?


GitHub link: https://github.com/apache/pulsar/discussions/22680


This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org

Re: [D] intermittent Messages loss while consuming the messages from consumer [pulsar]



GitHub user ragaur-tibco added a comment to the discussion: intermittent 
Messages loss while consuming the messages from consumer

@wallacepeng
I tried with two scenarios 
1. Running the producer and consumer and sending 1 message so that consumer 
able to consume that 1 message and then stop both producer and consumer then 
again start the producer and send 10k messages and stop the producer and run 
the consumer then saw that there are some message loss in while consuming from 
the consumer 
2. Running the consumer and producer and didn't send any message then stop the 
consumer and send 10k messages from the producer and then stop the producer and 
then start the consumer then saw there are some messages loss 

CC: @lhotari @visortelle 

can you please provide any cause of this? and also this issue is intermittent 

GitHub link: 
https://github.com/apache/pulsar/discussions/22681#discussioncomment-9359332


This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org

Re: [D] intermittent Messages loss while consuming the messages from consumer [pulsar]



GitHub user wallacepeng added a comment to the discussion: intermittent 
Messages loss while consuming the messages from consumer

messages should only be retained if there are subscriptions created . from your 
statement, u first publish messages, then start the consumer, i wonder if lot 
of messages already deleted ? 

GitHub link: 
https://github.com/apache/pulsar/discussions/22681#discussioncomment-9357169


This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org

Re: [PR] [fix][offload] Fix OOM in tiered storage, caused by unbounded offsets cache [pulsar]



hangc0276 commented on code in PR #22679:
URL: https://github.com/apache/pulsar/pull/22679#discussion_r1594228234


##
tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/OffsetsCache.java:
##
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.bookkeeper.mledger.offload.jcloud.impl;
+
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import io.grpc.netty.shaded.io.netty.util.concurrent.DefaultThreadFactory;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class OffsetsCache implements AutoCloseable {
+private static final int CACHE_TTL_SECONDS =
+
Integer.getInteger("pulsar.jclouds.readhandleimpl.offsetsscache.ttl.seconds", 
30 * 60);
+// limit the cache size to avoid OOM
+// 1 million entries consumes about 60MB of heap space
+private static final int CACHE_MAX_SIZE =
+
Integer.getInteger("pulsar.jclouds.readhandleimpl.offsetsscache.max.size", 
1_000_000);
+private final ScheduledExecutorService cacheEvictionExecutor;
+
+record Key(long ledgerId, long entryId) {
+
+}
+
+private final Cache entryOffsetsCache;
+
+public OffsetsCache() {
+if (CACHE_MAX_SIZE > 0) {
+entryOffsetsCache = CacheBuilder
+.newBuilder()
+.expireAfterAccess(CACHE_TTL_SECONDS, TimeUnit.SECONDS)
+.maximumSize(CACHE_MAX_SIZE)
+.build();
+cacheEvictionExecutor =
+Executors.newSingleThreadScheduledExecutor(
+new 
DefaultThreadFactory("jcloud-offsets-cache-eviction"));
+int period = Math.max(CACHE_TTL_SECONDS / 2, 1);
+cacheEvictionExecutor.scheduleAtFixedRate(() -> {
+entryOffsetsCache.cleanUp();
+}, period, period, TimeUnit.SECONDS);
+} else {
+cacheEvictionExecutor = null;
+entryOffsetsCache = null;
+}
+}
+
+public void put(long ledgerId, long entryId, long currentPosition) {
+if (entryOffsetsCache != null) {
+entryOffsetsCache.put(new Key(ledgerId, entryId), currentPosition);

Review Comment:
   It will create a lot of `Key` objects on the heap due to we build the offset 
index for each entry. I still have no idea how to optimize it.



##
tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/OffsetsCache.java:
##
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.bookkeeper.mledger.offload.jcloud.impl;
+
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import io.grpc.netty.shaded.io.netty.util.concurrent.DefaultThreadFactory;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class OffsetsCache implements AutoCloseable {
+private static final int CACHE_TTL_SECONDS =
+
Integer.getInteger("pulsar.jclouds.readhandleimpl.offsetsscache.ttl.seconds", 
30 * 60);
+// limit the cache size to avoid OOM
+// 1 million entries consumes about 60MB of heap space
+private static final int CACHE_MAX_SIZE =
+

Re: [D] consumer created multiple subscriptions wrongly [pulsar]



GitHub user wallacepeng closed a discussion: consumer created multiple 
subscriptions wrongly

one of our topics have strange behavior, the consumer can create multiple 
subscriptions , it is supposed to only create one single subsscription at a 
time 
```bin/pulsar-admin topics stats  persistent://1681/Marketo/marketo
{
  "msgRateIn" : 0.0,
  "msgThroughputIn" : 0.0,
  "msgRateOut" : 0.0,
  "msgThroughputOut" : 0.0,
  "bytesInCounter" : 0,
  "msgInCounter" : 0,
  "bytesOutCounter" : 0,
  "msgOutCounter" : 0,
  "averageMsgSize" : 0.0,
  "msgChunkPublished" : false,
  "storageSize" : 0,
  "backlogSize" : 0,
  "publishRateLimitedTimes" : 0,
  "earliestMsgPublishTimeInBacklogs" : 0,
  "offloadedStorageSize" : 0,
  "lastOffloadLedgerId" : 0,
  "lastOffloadSuccessTimeStamp" : 0,
  "lastOffloadFailureTimeStamp" : 0,
  "publishers" : [ ],
  "waitingPublishers" : 0,
  "subscriptions" : {
"load" : {
  "msgRateOut" : 0.0,
  "msgThroughputOut" : 0.0,
  "bytesOutCounter" : 0,
  "msgOutCounter" : 0,
  "msgRateRedeliver" : 0.0,
  "messageAckRate" : 0.0,
  "chunkedMessageRate" : 0,
  "msgBacklog" : 0,
  "backlogSize" : 0,
  "earliestMsgPublishTimeInBacklog" : 0,
  "msgBacklogNoDelayed" : 0,
  "blockedSubscriptionOnUnackedMsgs" : false,
  "msgDelayed" : 0,
  "unackedMessages" : 0,
  "type" : "Failover",
  "activeConsumerName" : "d9920",
  "msgRateExpired" : 0.0,
  "totalMsgExpired" : 0,
  "lastExpireTimestamp" : 0,
  "lastConsumedFlowTimestamp" : 1714765583341,
  "lastConsumedTimestamp" : 0,
  "lastAckedTimestamp" : 0,
  "lastMarkDeleteAdvancedTimestamp" : 0,
  "consumers" : [ {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesOutCounter" : 0,
"msgOutCounter" : 0,
"msgRateRedeliver" : 0.0,
"messageAckRate" : 0.0,
"chunkedMessageRate" : 0.0,
"consumerName" : "d9920",
"availablePermits" : 3,
"unackedMessages" : 0,
"avgMessagesPerEntry" : 0,
"blockedConsumerOnUnackedMsgs" : false,
"lastAckedTimestamp" : 0,
"lastConsumedTimestamp" : 0,
"metadata" : { },
"address" : "/10.1.170.77:50456",
"connectedSince" : "2024-05-03T19:44:37.996187Z",
"clientVersion" : "2.10.5.5"
  }, {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesOutCounter" : 0,
"msgOutCounter" : 0,
"msgRateRedeliver" : 0.0,
"messageAckRate" : 0.0,
"chunkedMessageRate" : 0.0,
"consumerName" : "92c71",
"availablePermits" : 3,
"unackedMessages" : 0,
"avgMessagesPerEntry" : 0,
"blockedConsumerOnUnackedMsgs" : false,
"lastAckedTimestamp" : 0,
"lastConsumedTimestamp" : 0,
"metadata" : { },
"address" : "/10.1.170.77:50456",
"connectedSince" : "2024-05-03T19:44:37.996345Z",
"clientVersion" : "2.10.5.5"
  }, {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesOutCounter" : 0,
"msgOutCounter" : 0,
"msgRateRedeliver" : 0.0,
"messageAckRate" : 0.0,
"chunkedMessageRate" : 0.0,
"consumerName" : "67e5b",
"availablePermits" : 3,
"unackedMessages" : 0,
"avgMessagesPerEntry" : 0,
"blockedConsumerOnUnackedMsgs" : false,
"lastAckedTimestamp" : 0,
"lastConsumedTimestamp" : 0,
"metadata" : { },
"address" : "/10.1.170.77:50446",
"connectedSince" : "2024-05-03T19:44:37.996506Z",
"clientVersion" : "2.10.5.5"
  }, {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesOutCounter" : 0,
"msgOutCounter" : 0,
"msgRateRedeliver" : 0.0,
"messageAckRate" : 0.0,
"chunkedMessageRate" : 0.0,
"consumerName" : "0bf65",
"availablePermits" : 3,
"unackedMessages" : 0,
"avgMessagesPerEntry" : 0,
"blockedConsumerOnUnackedMsgs" : false,
"lastAckedTimestamp" : 0,
"lastConsumedTimestamp" : 0,
"metadata" : { },
"address" : "/10.1.170.77:50420",
"connectedSince" : "2024-05-03T19:44:37.996604Z",
"clientVersion" : "2.10.5.5"
  }, {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesOutCounter" : 0,
"msgOutCounter" : 0,
"msgRateRedeliver" : 0.0,
"messageAckRate" : 0.0,
"chunkedMessageRate" : 0.0,
"consumerName" : "be9d9",
"availablePermits" : 3,
"unackedMessages" : 0,
"avgMessagesPerEntry" : 0,
"blockedConsumerOnUnackedMsgs" : false,
"lastAckedTimestamp" : 0,
"lastConsumedTimestamp" : 0,
"metadata" : { },
"address" : "/10.1.170.77:50456",
"connectedSince" : "2024-05-03T19:44:37.996681Z",
"clientVersion" : "2.10.5.5"
  }, {


GitHub link:

Re: [D] consumer created multiple subscriptions wrongly [pulsar]



GitHub user wallacepeng added a comment to the discussion: consumer created 
multiple subscriptions wrongly

it seems our code has a loop for odd reasons thought it looks correct . 

GitHub link: 
https://github.com/apache/pulsar/discussions/22648#discussioncomment-9355471


This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org

(pulsar) branch branch-2.10 updated: [fix] [broker] Fix nothing changed after removing dynamic configs (#22673)

This is an automated email from the ASF dual-hosted git repository.

yubiao pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/branch-2.10 by this push:
 new 02a8f8191e4 [fix] [broker] Fix nothing changed after removing dynamic 
configs (#22673)
02a8f8191e4 is described below

commit 02a8f8191e45ca576eb60743810a65a4ee780e72
Author: fengyubiao 
AuthorDate: Wed May 8 21:41:22 2024 +0800

[fix] [broker] Fix nothing changed after removing dynamic configs (#22673)

(cherry picked from commit ada31a96db9aabbb071f65229be746e61f954696)
---
 .../pulsar/broker/admin/impl/BrokersBase.java  |   8 +-
 .../pulsar/broker/service/BrokerService.java   | 140 +++--
 .../admin/AdminApiDynamicConfigurationsTest.java   |  18 +++
 3 files changed, 126 insertions(+), 40 deletions(-)

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java
index 5f73bc949de..57fdf6f2849 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java
@@ -230,7 +230,7 @@ public class BrokersBase extends AdminResource {
 @ApiResponse(code = 403, message = "You don't have admin 
permission to get configuration")})
 public List getDynamicConfigurationName() {
 validateSuperUserAccess();
-return BrokerService.getDynamicConfiguration();
+return pulsar().getBrokerService().getDynamicConfiguration();
 }
 
 @GET
@@ -253,11 +253,11 @@ public class BrokersBase extends AdminResource {
  */
 private synchronized CompletableFuture 
persistDynamicConfigurationAsync(
 String configName, String configValue) {
-if (!BrokerService.validateDynamicConfiguration(configName, 
configValue)) {
+if 
(!pulsar().getBrokerService().validateDynamicConfiguration(configName, 
configValue)) {
 return FutureUtil
 .failedFuture(new 
RestException(Status.PRECONDITION_FAILED, " Invalid dynamic-config value"));
 }
-if (BrokerService.isDynamicConfiguration(configName)) {
+if (pulsar().getBrokerService().isDynamicConfiguration(configName)) {
 return 
dynamicConfigurationResources().setDynamicConfigurationWithCreateAsync(old -> {
 Map configurationMap = 
old.orElseGet(Maps::newHashMap);
 configurationMap.put(configName, configValue);
@@ -451,7 +451,7 @@ public class BrokersBase extends AdminResource {
 }
 
 private CompletableFuture 
internalDeleteDynamicConfigurationOnMetadataAsync(String configName) {
-if (!BrokerService.isDynamicConfiguration(configName)) {
+if (!pulsar().getBrokerService().isDynamicConfiguration(configName)) {
 throw new RestException(Status.PRECONDITION_FAILED, " Can't update 
non-dynamic configuration");
 } else {
 return 
dynamicConfigurationResources().setDynamicConfigurationAsync(old -> {
diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
index 4b86df9e379..f6103dbae76 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
@@ -107,6 +107,7 @@ import org.apache.pulsar.broker.intercept.BrokerInterceptor;
 import org.apache.pulsar.broker.intercept.ManagedLedgerInterceptorImpl;
 import org.apache.pulsar.broker.loadbalance.LoadManager;
 import org.apache.pulsar.broker.namespace.NamespaceService;
+import org.apache.pulsar.broker.resources.DynamicConfigurationResources;
 import org.apache.pulsar.broker.resources.LocalPoliciesResources;
 import org.apache.pulsar.broker.resources.NamespaceResources;
 import 
org.apache.pulsar.broker.resources.NamespaceResources.PartitionedTopicResources;
@@ -220,8 +221,7 @@ public class BrokerService implements Closeable {
 private final OrderedExecutor topicOrderedExecutor;
 // offline topic backlog cache
 private final ConcurrentOpenHashMap offlineTopicStatCache;
-private static final ConcurrentOpenHashMap 
dynamicConfigurationMap =
-prepareDynamicConfigurationMap();
+private final ConcurrentOpenHashMap 
dynamicConfigurationMap;
 private final ConcurrentOpenHashMap> 
configRegisteredListeners;
 
 private final ConcurrentLinkedQueue 
pendingTopicLoadingQueue;
@@ -289,6 +289,7 @@ public class BrokerService implements Closeable {
 
 public BrokerService(PulsarService pulsar, EventLoopGroup eventLoopGroup) 
throws Exception {
 this.pulsar = pulsar;
+this.dynamicConfigurationMap = prepareDynamicConfigurationMap();

(pulsar) branch branch-2.11 updated: [fix] [broker] Fix nothing changed after removing dynamic configs (#22673)

This is an automated email from the ASF dual-hosted git repository.

yubiao pushed a commit to branch branch-2.11
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/branch-2.11 by this push:
 new 28680d330fc [fix] [broker] Fix nothing changed after removing dynamic 
configs (#22673)
28680d330fc is described below

commit 28680d330fc2dec34eff2a8fdcec8ba13395
Author: fengyubiao 
AuthorDate: Wed May 8 21:41:22 2024 +0800

[fix] [broker] Fix nothing changed after removing dynamic configs (#22673)

(cherry picked from commit ada31a96db9aabbb071f65229be746e61f954696)
---
 .../pulsar/broker/admin/impl/BrokersBase.java  |   8 +-
 .../pulsar/broker/service/BrokerService.java   | 126 +++--
 .../admin/AdminApiDynamicConfigurationsTest.java   |  40 +++
 3 files changed, 135 insertions(+), 39 deletions(-)

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java
index ac730c01eab..df32c64b88c 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java
@@ -254,7 +254,7 @@ public class BrokersBase extends AdminResource {
 @ApiResponse(code = 403, message = "You don't have admin 
permission to get configuration")})
 public void getDynamicConfigurationName(@Suspended AsyncResponse 
asyncResponse) {
 validateSuperUserAccessAsync()
-.thenAccept(__ -> 
asyncResponse.resume(BrokerService.getDynamicConfiguration()))
+.thenAccept(__ -> 
asyncResponse.resume(pulsar().getBrokerService().getDynamicConfiguration()))
 .exceptionally(ex -> {
 LOG.error("[{}] Failed to get all dynamic configuration 
names.", clientAppId(), ex);
 resumeAsyncResponseExceptionally(asyncResponse, ex);
@@ -287,11 +287,11 @@ public class BrokersBase extends AdminResource {
  */
 private synchronized CompletableFuture 
persistDynamicConfigurationAsync(
 String configName, String configValue) {
-if (!BrokerService.validateDynamicConfiguration(configName, 
configValue)) {
+if 
(!pulsar().getBrokerService().validateDynamicConfiguration(configName, 
configValue)) {
 return FutureUtil
 .failedFuture(new 
RestException(Status.PRECONDITION_FAILED, " Invalid dynamic-config value"));
 }
-if (BrokerService.isDynamicConfiguration(configName)) {
+if (pulsar().getBrokerService().isDynamicConfiguration(configName)) {
 return 
dynamicConfigurationResources().setDynamicConfigurationWithCreateAsync(old -> {
 Map configurationMap = 
old.orElseGet(Maps::newHashMap);
 configurationMap.put(configName, configValue);
@@ -512,7 +512,7 @@ public class BrokersBase extends AdminResource {
 }
 
 private CompletableFuture 
internalDeleteDynamicConfigurationOnMetadataAsync(String configName) {
-if (!BrokerService.isDynamicConfiguration(configName)) {
+if (!pulsar().getBrokerService().isDynamicConfiguration(configName)) {
 throw new RestException(Status.PRECONDITION_FAILED, " Can't update 
non-dynamic configuration");
 } else {
 return 
dynamicConfigurationResources().setDynamicConfigurationAsync(old -> {
diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
index f05463a128c..090d5ce0b54 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
@@ -229,8 +229,7 @@ public class BrokerService implements Closeable {
 private final OrderedExecutor topicOrderedExecutor;
 // offline topic backlog cache
 private final ConcurrentOpenHashMap offlineTopicStatCache;
-private static final ConcurrentOpenHashMap 
dynamicConfigurationMap =
-prepareDynamicConfigurationMap();
+private final ConcurrentOpenHashMap 
dynamicConfigurationMap;
 private final ConcurrentOpenHashMap> 
configRegisteredListeners;
 
 private final ConcurrentLinkedQueue 
pendingTopicLoadingQueue;
@@ -299,6 +298,7 @@ public class BrokerService implements Closeable {
 
 public BrokerService(PulsarService pulsar, EventLoopGroup eventLoopGroup) 
throws Exception {
 this.pulsar = pulsar;
+this.dynamicConfigurationMap = prepareDynamicConfigurationMap();
 this.preciseTopicPublishRateLimitingEnable =
 
pulsar.getConfiguration().isPreciseTopicPublishRateLimiterEnable();
 this.managedLedgerFactory = pulsar.getManagedLedgerFactory();
@@ -2310,39 +2310,75 @@ public

(pulsar) branch branch-3.0 updated: [fix] [broker] Fix nothing changed after removing dynamic configs (#22673)

This is an automated email from the ASF dual-hosted git repository.

yubiao pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/branch-3.0 by this push:
 new f2c43675f69 [fix] [broker] Fix nothing changed after removing dynamic 
configs (#22673)
f2c43675f69 is described below

commit f2c43675f699828e677f7d21b13f6cf1bd48dd23
Author: fengyubiao 
AuthorDate: Wed May 8 21:41:22 2024 +0800

[fix] [broker] Fix nothing changed after removing dynamic configs (#22673)

(cherry picked from commit ada31a96db9aabbb071f65229be746e61f954696)
---
 .../pulsar/broker/admin/impl/BrokersBase.java  |   8 +-
 .../pulsar/broker/service/BrokerService.java   | 138 +++--
 .../admin/AdminApiDynamicConfigurationsTest.java   |  40 ++
 3 files changed, 141 insertions(+), 45 deletions(-)

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java
index 57650758bbc..ea793cabff5 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java
@@ -264,7 +264,7 @@ public class BrokersBase extends AdminResource {
 @ApiResponse(code = 403, message = "You don't have admin 
permission to get configuration")})
 public void getDynamicConfigurationName(@Suspended AsyncResponse 
asyncResponse) {
 validateSuperUserAccessAsync()
-.thenAccept(__ -> 
asyncResponse.resume(BrokerService.getDynamicConfiguration()))
+.thenAccept(__ -> 
asyncResponse.resume(pulsar().getBrokerService().getDynamicConfiguration()))
 .exceptionally(ex -> {
 LOG.error("[{}] Failed to get all dynamic configuration 
names.", clientAppId(), ex);
 resumeAsyncResponseExceptionally(asyncResponse, ex);
@@ -297,11 +297,11 @@ public class BrokersBase extends AdminResource {
  */
 private synchronized CompletableFuture 
persistDynamicConfigurationAsync(
 String configName, String configValue) {
-if (!BrokerService.validateDynamicConfiguration(configName, 
configValue)) {
+if 
(!pulsar().getBrokerService().validateDynamicConfiguration(configName, 
configValue)) {
 return FutureUtil
 .failedFuture(new 
RestException(Status.PRECONDITION_FAILED, " Invalid dynamic-config value"));
 }
-if (BrokerService.isDynamicConfiguration(configName)) {
+if (pulsar().getBrokerService().isDynamicConfiguration(configName)) {
 return 
dynamicConfigurationResources().setDynamicConfigurationWithCreateAsync(old -> {
 Map configurationMap = 
old.orElseGet(Maps::newHashMap);
 configurationMap.put(configName, configValue);
@@ -526,7 +526,7 @@ public class BrokersBase extends AdminResource {
 }
 
 private CompletableFuture 
internalDeleteDynamicConfigurationOnMetadataAsync(String configName) {
-if (!BrokerService.isDynamicConfiguration(configName)) {
+if (!pulsar().getBrokerService().isDynamicConfiguration(configName)) {
 throw new RestException(Status.PRECONDITION_FAILED, " Can't update 
non-dynamic configuration");
 } else {
 return 
dynamicConfigurationResources().setDynamicConfigurationAsync(old -> {
diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
index 2adacecb1fa..bcfe531e39f 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
@@ -218,8 +218,7 @@ public class BrokerService implements Closeable {
 private final OrderedExecutor topicOrderedExecutor;
 // offline topic backlog cache
 private final ConcurrentOpenHashMap offlineTopicStatCache;
-private static final ConcurrentOpenHashMap 
dynamicConfigurationMap =
-prepareDynamicConfigurationMap();
+private final ConcurrentOpenHashMap 
dynamicConfigurationMap;
 private final ConcurrentOpenHashMap> 
configRegisteredListeners;
 
 private final ConcurrentLinkedQueue 
pendingTopicLoadingQueue;
@@ -292,6 +291,7 @@ public class BrokerService implements Closeable {
 
 public BrokerService(PulsarService pulsar, EventLoopGroup eventLoopGroup) 
throws Exception {
 this.pulsar = pulsar;
+this.dynamicConfigurationMap = prepareDynamicConfigurationMap();
 this.preciseTopicPublishRateLimitingEnable =
 
pulsar.getConfiguration().isPreciseTopicPublishRateLimiterEnable();
 this.managedLedgerFactory = pulsar.getManagedLedgerFactory();
@@ -2517,37 +2517,73 @@ public

Re: [I] [Bug] parseMessageMetadata error when broker entry metadata enable with high loading [pulsar]



semistone commented on issue #22601:
URL: https://github.com/apache/pulsar/issues/22601#issuecomment-2100631193

   I almost could reproduce by perf tool
   when very few payload > 30K bytes. others are 3K bytes
   then
   error happen when messageKeyGenerationMode=random
   if without messageKeyGenerationMode, then error disappear
   
   I guess in batch mode, payload size have some restriction.
   
   let me confirm again tomorrow to make sure I didn't make any stupid mistake 
during my test.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(pulsar) branch branch-3.2 updated: [fix] [broker] Fix nothing changed after removing dynamic configs (#22673)

This is an automated email from the ASF dual-hosted git repository.

yubiao pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/branch-3.2 by this push:
 new c33fdeab2fe [fix] [broker] Fix nothing changed after removing dynamic 
configs (#22673)
c33fdeab2fe is described below

commit c33fdeab2fedab9b0f62b612fe14032eef5ba0f6
Author: fengyubiao 
AuthorDate: Wed May 8 21:41:22 2024 +0800

[fix] [broker] Fix nothing changed after removing dynamic configs (#22673)

(cherry picked from commit ada31a96db9aabbb071f65229be746e61f954696)
---
 .../pulsar/broker/service/BrokerService.java   | 137 ++---
 .../admin/AdminApiDynamicConfigurationsTest.java   |  68 ++
 2 files changed, 160 insertions(+), 45 deletions(-)

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
index 60d56c0d908..6d10510dee2 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
@@ -219,8 +219,7 @@ public class BrokerService implements Closeable {
 private final OrderedExecutor topicOrderedExecutor;
 // offline topic backlog cache
 private final ConcurrentOpenHashMap offlineTopicStatCache;
-private final ConcurrentOpenHashMap 
dynamicConfigurationMap =
-prepareDynamicConfigurationMap();
+private final ConcurrentOpenHashMap 
dynamicConfigurationMap;
 private final ConcurrentOpenHashMap> 
configRegisteredListeners;
 
 private final ConcurrentLinkedQueue 
pendingTopicLoadingQueue;
@@ -292,6 +291,7 @@ public class BrokerService implements Closeable {
 
 public BrokerService(PulsarService pulsar, EventLoopGroup eventLoopGroup) 
throws Exception {
 this.pulsar = pulsar;
+this.dynamicConfigurationMap = prepareDynamicConfigurationMap();
 this.brokerPublishRateLimiter = new 
PublishRateLimiterImpl(pulsar.getMonotonicSnapshotClock());
 this.preciseTopicPublishRateLimitingEnable =
 
pulsar.getConfiguration().isPreciseTopicPublishRateLimiterEnable();
@@ -2434,40 +2434,71 @@ public class BrokerService implements Closeable {
 
 if (dynamicConfigResources != null) {
 dynamicConfigResources.getDynamicConfigurationAsync()
-.thenAccept(optMap -> {
-if (!optMap.isPresent()) {
-return;
+.thenAccept(optMap -> {
+// Case some dynamic configs have been removed.
+dynamicConfigurationMap.forEach((configKey, fieldWrapper) 
-> {
+boolean configRemoved = optMap.isEmpty() || 
!optMap.get().containsKey(configKey);
+if (fieldWrapper.lastDynamicValue != null && 
configRemoved) {
+configValueChanged(configKey, null);
 }
-Map data = optMap.get();
-data.forEach((configKey, value) -> {
-ConfigField configFieldWrapper = 
dynamicConfigurationMap.get(configKey);
-if (configFieldWrapper == null) {
-log.warn("{} does not exist in 
dynamicConfigurationMap, skip this config.", configKey);
-return;
-}
-Field configField = configFieldWrapper.field;
-Consumer listener = 
configRegisteredListeners.get(configKey);
-try {
-final Object existingValue;
-final Object newValue;
-if (configField != null) {
-newValue = 
FieldParser.value(data.get(configKey), configField);
-existingValue = 
configField.get(pulsar.getConfiguration());
-configField.set(pulsar.getConfiguration(), 
newValue);
-} else {
-newValue = value;
-existingValue = 
configFieldWrapper.customValue;
-configFieldWrapper.customValue = newValue 
== null ? null : String.valueOf(newValue);
-}
-log.info("Successfully updated configuration 
{}/{}", configKey, data.get(configKey));
-if (listener != null && 
!Objects.equals(existingValue, newValue)) {
-listener.accept(newValue);
-}
-} catch (Exception e) {
-

(pulsar) branch master updated: [fix] [broker] Fix nothing changed after removing dynamic configs (#22673)

This is an automated email from the ASF dual-hosted git repository.

yubiao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new ada31a96db9 [fix] [broker] Fix nothing changed after removing dynamic 
configs (#22673)
ada31a96db9 is described below

commit ada31a96db9aabbb071f65229be746e61f954696
Author: fengyubiao 
AuthorDate: Wed May 8 21:41:22 2024 +0800

[fix] [broker] Fix nothing changed after removing dynamic configs (#22673)
---
 .../pulsar/broker/service/BrokerService.java   | 137 ++---
 .../admin/AdminApiDynamicConfigurationsTest.java   |  68 ++
 2 files changed, 160 insertions(+), 45 deletions(-)

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
index 6e23deaa6fa..c1b2b9e1da9 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
@@ -229,8 +229,7 @@ public class BrokerService implements Closeable {
 private final OrderedExecutor topicOrderedExecutor;
 // offline topic backlog cache
 private final ConcurrentOpenHashMap offlineTopicStatCache;
-private final ConcurrentOpenHashMap 
dynamicConfigurationMap =
-prepareDynamicConfigurationMap();
+private final ConcurrentOpenHashMap 
dynamicConfigurationMap;
 private final ConcurrentOpenHashMap> 
configRegisteredListeners;
 
 private final ConcurrentLinkedQueue 
pendingTopicLoadingQueue;
@@ -313,6 +312,7 @@ public class BrokerService implements Closeable {
 
 public BrokerService(PulsarService pulsar, EventLoopGroup eventLoopGroup) 
throws Exception {
 this.pulsar = pulsar;
+this.dynamicConfigurationMap = prepareDynamicConfigurationMap();
 this.brokerPublishRateLimiter = new 
PublishRateLimiterImpl(pulsar.getMonotonicSnapshotClock());
 this.preciseTopicPublishRateLimitingEnable =
 
pulsar.getConfiguration().isPreciseTopicPublishRateLimiterEnable();
@@ -2496,40 +2496,71 @@ public class BrokerService implements Closeable {
 
 if (dynamicConfigResources != null) {
 dynamicConfigResources.getDynamicConfigurationAsync()
-.thenAccept(optMap -> {
-if (!optMap.isPresent()) {
-return;
+.thenAccept(optMap -> {
+// Case some dynamic configs have been removed.
+dynamicConfigurationMap.forEach((configKey, fieldWrapper) 
-> {
+boolean configRemoved = optMap.isEmpty() || 
!optMap.get().containsKey(configKey);
+if (fieldWrapper.lastDynamicValue != null && 
configRemoved) {
+configValueChanged(configKey, null);
 }
-Map data = optMap.get();
-data.forEach((configKey, value) -> {
-ConfigField configFieldWrapper = 
dynamicConfigurationMap.get(configKey);
-if (configFieldWrapper == null) {
-log.warn("{} does not exist in 
dynamicConfigurationMap, skip this config.", configKey);
-return;
-}
-Field configField = configFieldWrapper.field;
-Consumer listener = 
configRegisteredListeners.get(configKey);
-try {
-final Object existingValue;
-final Object newValue;
-if (configField != null) {
-newValue = 
FieldParser.value(data.get(configKey), configField);
-existingValue = 
configField.get(pulsar.getConfiguration());
-configField.set(pulsar.getConfiguration(), 
newValue);
-} else {
-newValue = value;
-existingValue = 
configFieldWrapper.customValue;
-configFieldWrapper.customValue = newValue 
== null ? null : String.valueOf(newValue);
-}
-log.info("Successfully updated configuration 
{}/{}", configKey, data.get(configKey));
-if (listener != null && 
!Objects.equals(existingValue, newValue)) {
-listener.accept(newValue);
-}
-} catch (Exception e) {
-log.error("Failed to update config {}", 
configKey, e);
-

Re: [PR] [fix] [broker] Fix nothing changed after removing dynamic configs [pulsar]



poorbarcode merged PR #22673:
URL: https://github.com/apache/pulsar/pull/22673


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] [fix][offload] Fix OOM in tiered storage, caused by unbounded offsets cache [pulsar]



lhotari opened a new pull request, #22679:
URL: https://github.com/apache/pulsar/pull/22679

   ### Motivation
   
   In #16417 a cache was added for caching offsets. The problem with this 
implementation is that in some cases, the 
   memory consumption can go very high and cause OOM.
   
   ### Modifications
   
   - Move the cache to the 
org.apache.bookkeeper.mledger.offload.jcloud.JCloudLedgerOffloaderFactory level 
and share it across all offloaded ledgers.
   - Add a default maximum size of 1M entries. This consumes about 60MB of heap 
at maximum.
 - add system property 
`pulsar.jclouds.readhandleimpl.offsetsscache.max.size` to adjust this, however 
there shouldn't be a need to tune it.
   - Add scheduler to run the cleanup job periodically so that the TTL limit 
will be enforced in all cases. 
   
   ### Documentation
   
   
   
   - [ ] `doc` 
   - [ ] `doc-required` 
   - [x] `doc-not-needed` 
   - [ ] `doc-complete` 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[I] Add parameters for KeyStore TLS in Pulsar test client [pulsar]



shasank112001 opened a new issue, #22678:
URL: https://github.com/apache/pulsar/issues/22678

   ### Search before asking
   
   - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) 
and found nothing similar.
   
   
   ### Motivation
   
   If you want to create a TestClient which uses AuthenticationKeyStoreTls as 
its authPlugin, Pulsar Test Admin/Client utilized in Pulsar-perf is unable to 
setup a vaild SSL context due to the requirement of the property 
"useKeyStoreTls" to be "true" for using keystores properly.
   Moreover, utilizing the property "useKeyStoreTls" requires the use of 
trust-store and not trust certificates, therefore requiring additional 
trustStoreType, trustStorePath and trustStorePass as parameters to be 
available, to utilize PulsarPerf.
   
   The main reason to require this change is the following code in 
AsyncHttpConnector 
   ```java
   if (conf.isUseKeyStoreTls()) {
   KeyStoreParams params = authData.hasDataForTls() ? 
authData.getTlsKeyStoreParams() :
   new KeyStoreParams(conf.getTlsKeyStoreType(), 
conf.getTlsKeyStorePath(),
   conf.getTlsKeyStorePassword());
   
   final SSLContext sslCtx = 
KeyStoreSSLContext.createClientSslContext(
   conf.getSslProvider(),
   params.getKeyStoreType(),
   params.getKeyStorePath(),
   params.getKeyStorePassword(),
   conf.isTlsAllowInsecureConnection(),
   conf.getTlsTrustStoreType(),
   conf.getTlsTrustStorePath(),
   conf.getTlsTrustStorePassword(),
   conf.getTlsCiphers(),
   conf.getTlsProtocols());
   
   JsseSslEngineFactory sslEngineFactory = new 
JsseSslEngineFactory(sslCtx);
   confBuilder.setSslEngineFactory(sslEngineFactory);
   } 
   ```
   
   I understand that users can just use AuthenticationTls instead of 
AuthenticationKeyStoreTls for Performance testing as it will work with the mTLS 
authentication scheme, but my motivation to provide these arguments is to allow 
for custom authentication/authorization providers that utilize keystores and 
want to perform performance testing using Pulsar Perf.
   
   ### Solution
   
   Create new Arguments for the following properties, in 
PerformanceBaseArguments.java :
   1. useKeyStoreTls
   2. trustStoreType
   3. trustStorePath
   4. trustStorePass
   5. keyStoreType
   6. keyStorePath
   7. keyStorePass
   
   Update the code to change between TrustCerts and TrustStore based on 
useKeyStoreTls.
   
   ### Alternatives
   
   An alternative solution is to update AuthenticationDataProviders that 
utilize keystores to override functions to provide trustStore as stream and the 
private key as well, not depending on the "useKeyStoreTls" property, but as we 
already have properties for trustStoreType, trustStorePath and trustStorePass 
which are only invoked when useKeyStoreTls is true, it makes more sense to 
expose these properties as parameters in Perf Clients.
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [X] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [fix][ml] Fix entryCache potential thread-safety issue [pulsar]



dao-jun closed pull request #22677: [fix][ml] Fix entryCache potential 
thread-safety issue
URL: https://github.com/apache/pulsar/pull/22677


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] [fix][ml] Fix entryCache potential thread-safety issue [pulsar]



dao-jun opened a new pull request, #22677:
URL: https://github.com/apache/pulsar/pull/22677

   ### Motivation
   
   We pass dataBuffer to callbacks after add entry to entryCache
   
https://github.com/apache/pulsar/blob/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java#L719-L736
   
   If we do something with the dataBuffer(parse metadata, eg...), and at the 
same time, there is a consumer read the entry from entryCache and parse 
metadata, it may lead to parse metadata exception.
   
   ### Modifications
   
   
   
   ### Verifying this change
   
   - [ ] Make sure that the change passes the CI checks.
   
   *(Please pick either of the following options)*
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   *(or)*
   
   This change is already covered by existing tests, such as *(please describe 
tests)*.
   
   *(or)*
   
   This change added tests and can be verified as follows:
   
   *(example:)*
 - *Added integration tests for end-to-end deployment with large payloads 
(10MB)*
 - *Extended integration test for recovery after broker failure*
   
   ### Does this pull request potentially affect one of the following parts:
   
   
   
   *If the box was checked, please highlight the changes*
   
   - [ ] Dependencies (add or upgrade a dependency)
   - [ ] The public API
   - [ ] The schema
   - [ ] The default values of configurations
   - [ ] The threading model
   - [ ] The binary protocol
   - [ ] The REST endpoints
   - [ ] The admin CLI options
   - [ ] The metrics
   - [ ] Anything that affects deployment
   
   ### Documentation
   
   
   
   - [ ] `doc` 
   - [ ] `doc-required` 
   - [x] `doc-not-needed` 
   - [ ] `doc-complete` 
   
   ### Matching PR in forked repository
   
   PR in forked repository: 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] [fix][ml] Remove duplicated field initialization of ML [pulsar]



dao-jun opened a new pull request, #22676:
URL: https://github.com/apache/pulsar/pull/22676

   ### Motivation
   
   `managedLedgerInterceptor` of `ManagedLedgerImpl` initialized twice, remove 
one
   
   ### Modifications
   
   
   
   ### Verifying this change
   
   - [ ] Make sure that the change passes the CI checks.
   
   *(Please pick either of the following options)*
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   *(or)*
   
   This change is already covered by existing tests, such as *(please describe 
tests)*.
   
   *(or)*
   
   This change added tests and can be verified as follows:
   
   *(example:)*
 - *Added integration tests for end-to-end deployment with large payloads 
(10MB)*
 - *Extended integration test for recovery after broker failure*
   
   ### Does this pull request potentially affect one of the following parts:
   
   
   
   *If the box was checked, please highlight the changes*
   
   - [ ] Dependencies (add or upgrade a dependency)
   - [ ] The public API
   - [ ] The schema
   - [ ] The default values of configurations
   - [ ] The threading model
   - [ ] The binary protocol
   - [ ] The REST endpoints
   - [ ] The admin CLI options
   - [ ] The metrics
   - [ ] Anything that affects deployment
   
   ### Documentation
   
   
   
   - [ ] `doc` 
   - [ ] `doc-required` 
   - [x] `doc-not-needed` 
   - [ ] `doc-complete` 
   
   ### Matching PR in forked repository
   
   PR in forked repository: 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(pulsar) branch master updated: [fix] [broker] rename to changeMaxReadPositionCount (#22656)

2024-05-08 Thread daojun

This is an automated email from the ASF dual-hosted git repository.

daojun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 5ab05129514 [fix] [broker] rename to changeMaxReadPositionCount 
(#22656)
5ab05129514 is described below

commit 5ab05129514c1e71a09ec3f28b2b2dda9ce3e47f
Author: Wenzhi Feng <52550727+thetumb...@users.noreply.github.com>
AuthorDate: Wed May 8 19:34:00 2024 +0800

[fix] [broker] rename to changeMaxReadPositionCount (#22656)
---
 .../transaction/buffer/impl/TopicTransactionBuffer.java  | 16 
 .../pulsar/broker/transaction/TransactionTest.java   | 12 ++--
 2 files changed, 14 insertions(+), 14 deletions(-)

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/TopicTransactionBuffer.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/TopicTransactionBuffer.java
index a36216bd625..81c9ecfc728 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/TopicTransactionBuffer.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/TopicTransactionBuffer.java
@@ -76,8 +76,8 @@ public class TopicTransactionBuffer extends 
TopicTransactionBufferState implemen
  */
 private final LinkedMap ongoingTxns = new 
LinkedMap<>();
 
-// when add abort or change max read position, the count will +1. Take 
snapshot will set 0 into it.
-private final AtomicLong changeMaxReadPositionAndAddAbortTimes = new 
AtomicLong();
+// when change max read position, the count will +1. Take snapshot will 
reset the count.
+private final AtomicLong changeMaxReadPositionCount = new AtomicLong();
 
 private final LongAdder txnCommittedCounter = new LongAdder();
 
@@ -429,15 +429,15 @@ public class TopicTransactionBuffer extends 
TopicTransactionBufferState implemen
 }
 
 private void takeSnapshotByChangeTimes() {
-if (changeMaxReadPositionAndAddAbortTimes.get() >= 
takeSnapshotIntervalNumber) {
-this.changeMaxReadPositionAndAddAbortTimes.set(0);
+if (changeMaxReadPositionCount.get() >= takeSnapshotIntervalNumber) {
+this.changeMaxReadPositionCount.set(0);
 
this.snapshotAbortedTxnProcessor.takeAbortedTxnsSnapshot(this.maxReadPosition);
 }
 }
 
 private void takeSnapshotByTimeout() {
-if (changeMaxReadPositionAndAddAbortTimes.get() > 0) {
-this.changeMaxReadPositionAndAddAbortTimes.set(0);
+if (changeMaxReadPositionCount.get() > 0) {
+this.changeMaxReadPositionCount.set(0);
 
this.snapshotAbortedTxnProcessor.takeAbortedTxnsSnapshot(this.maxReadPosition);
 }
 this.timer.newTimeout(TopicTransactionBuffer.this,
@@ -454,7 +454,7 @@ public class TopicTransactionBuffer extends 
TopicTransactionBufferState implemen
 maxReadPosition = (PositionImpl) 
topic.getManagedLedger().getLastConfirmedEntry();
 }
 if (preMaxReadPosition.compareTo(this.maxReadPosition) != 0) {
-this.changeMaxReadPositionAndAddAbortTimes.getAndIncrement();
+this.changeMaxReadPositionCount.getAndIncrement();
 }
 }
 
@@ -489,7 +489,7 @@ public class TopicTransactionBuffer extends 
TopicTransactionBufferState implemen
 } else if (checkIfReady()) {
 if (ongoingTxns.isEmpty()) {
 maxReadPosition = position;
-changeMaxReadPositionAndAddAbortTimes.incrementAndGet();
+changeMaxReadPositionCount.incrementAndGet();
 }
 }
 }
diff --git 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TransactionTest.java
 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TransactionTest.java
index e8c15d193a2..5e806bb9cee 100644
--- 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TransactionTest.java
+++ 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TransactionTest.java
@@ -1095,10 +1095,10 @@ public class TransactionTest extends 
TransactionTestBase {
 }
 
 @Test
-public void 
testNotChangeMaxReadPositionAndAddAbortTimesWhenCheckIfNoSnapshot() throws 
Exception {
+public void testNotChangeMaxReadPositionCountWhenCheckIfNoSnapshot() 
throws Exception {
 PersistentTopic persistentTopic = (PersistentTopic) 
getPulsarServiceList().get(0)
 .getBrokerService()
-.getTopic(NAMESPACE1 + 
"/changeMaxReadPositionAndAddAbortTimes" + UUID.randomUUID(), true)
+.getTopic(NAMESPACE1 + "/changeMaxReadPositionCount" + 
UUID.randomUUID(), true)
 .get().get();
 TransactionBuffer buffer = persistentTopic.getTransactionBuffer();
 Field processorField =

Re: [PR] [fix] [broker] rename to changeMaxReadPositionCount [pulsar]



dao-jun merged PR #22656:
URL: https://github.com/apache/pulsar/pull/22656


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [fix] [broker] Fix nothing changed after removing dynamic configs [pulsar]



poorbarcode commented on code in PR #22673:
URL: https://github.com/apache/pulsar/pull/22673#discussion_r1593844969


##
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java:
##
@@ -2488,40 +2488,71 @@ private void handleDynamicConfigurationUpdates() {
 
 if (dynamicConfigResources != null) {
 dynamicConfigResources.getDynamicConfigurationAsync()
-.thenAccept(optMap -> {
-if (!optMap.isPresent()) {
-return;
+.thenAccept(optMap -> {
+// Case some dynamic configs have been removed.
+dynamicConfigurationMap.forEach((configKey, fieldWrapper) 
-> {
+boolean configRemoved = optMap.isEmpty() || 
!optMap.get().containsKey(configKey);
+if (fieldWrapper.lastDynamicValue != null && 
configRemoved) {
+configValueChanged(configKey, null);
 }
-Map data = optMap.get();
-data.forEach((configKey, value) -> {
-ConfigField configFieldWrapper = 
dynamicConfigurationMap.get(configKey);
-if (configFieldWrapper == null) {
-log.warn("{} does not exist in 
dynamicConfigurationMap, skip this config.", configKey);
-return;
-}
-Field configField = configFieldWrapper.field;
-Consumer listener = 
configRegisteredListeners.get(configKey);
-try {
-final Object existingValue;
-final Object newValue;
-if (configField != null) {
-newValue = 
FieldParser.value(data.get(configKey), configField);
-existingValue = 
configField.get(pulsar.getConfiguration());
-configField.set(pulsar.getConfiguration(), 
newValue);
-} else {
-newValue = value;
-existingValue = 
configFieldWrapper.customValue;
-configFieldWrapper.customValue = newValue 
== null ? null : String.valueOf(newValue);
-}
-log.info("Successfully updated configuration 
{}/{}", configKey, data.get(configKey));
-if (listener != null && 
!Objects.equals(existingValue, newValue)) {
-listener.accept(newValue);
-}
-} catch (Exception e) {
-log.error("Failed to update config {}", 
configKey, e);
-}
-});
 });
+// Some configs have been changed.
+if (!optMap.isPresent()) {
+return;
+}
+Map data = optMap.get();
+data.forEach((configKey, value) -> {
+configValueChanged(configKey, value);
+});
+});
+}
+}
+
+private void configValueChanged(String configKey, String newValueStr) {
+ConfigField configFieldWrapper = 
dynamicConfigurationMap.get(configKey);
+if (configFieldWrapper == null) {
+log.warn("{} does not exist in dynamicConfigurationMap, skip this 
config.", configKey);
+return;
+}
+Consumer listener = configRegisteredListeners.get(configKey);
+try {
+// Convert existingValue and newValue.
+final Object existingValue;
+final Object newValue;
+if (configFieldWrapper.field != null) {
+if (StringUtils.isBlank(newValueStr)) {
+newValue = configFieldWrapper.defaultValue;
+} else {
+newValue = FieldParser.value(newValueStr, 
configFieldWrapper.field);
+}
+existingValue = 
configFieldWrapper.field.get(pulsar.getConfiguration());
+configFieldWrapper.field.set(pulsar.getConfiguration(), 
newValue);
+} else {
+// This case only occurs when it is a customized item.
+// See: 
https://github.com/apache/pulsar/blob/master/pip/pip-300.md.
+log.info("k:{} v:{} Skip to update configuration in memory 
because it is a customized item, only"
++ " trigger an event listeners.", configKey, 
newValueStr);

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message,

[PR] [cleanup] Remove unnecessary script and update documentation [pulsar]



crossoverJie opened a new pull request, #22675:
URL: https://github.com/apache/pulsar/pull/22675

   
   
   ### Modifications
   
   Remove unnecessary script and update documentation.
   
   ### Verifying this change
   
   - [x] Make sure that the change passes the CI checks.
   
   *(Please pick either of the following options)*
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   *(or)*
   
   This change is already covered by existing tests, such as *(please describe 
tests)*.
   
   *(or)*
   
   This change added tests and can be verified as follows:
   
   *(example:)*
 - *Added integration tests for end-to-end deployment with large payloads 
(10MB)*
 - *Extended integration test for recovery after broker failure*
   
   ### Does this pull request potentially affect one of the following parts:
   
   
   
   *If the box was checked, please highlight the changes*
   
   - [ ] Dependencies (add or upgrade a dependency)
   - [ ] The public API
   - [ ] The schema
   - [ ] The default values of configurations
   - [ ] The threading model
   - [ ] The binary protocol
   - [ ] The REST endpoints
   - [ ] The admin CLI options
   - [ ] The metrics
   - [ ] Anything that affects deployment
   
   ### Documentation
   
   
   - [ ] `doc` 
   - [ ] `doc-required` 
   - [x] `doc-not-needed` 
   - [ ] `doc-complete` 
   
   ### Matching PR in forked repository
   
   PR in forked repository: https://github.com/crossoverJie/pulsar/pull/25
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(pulsar) branch branch-3.2 updated: Revert "[fix][sec] Upgrade Debezium oracle connector version to avoid… (#22668)

This is an automated email from the ASF dual-hosted git repository.

technoboy pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/branch-3.2 by this push:
 new f7984d74d19 Revert "[fix][sec] Upgrade Debezium oracle connector 
version to avoid… (#22668)
f7984d74d19 is described below

commit f7984d74d19e50d31f7ea2abacef8430e4cf95bd
Author: Lari Hotari 
AuthorDate: Wed May 8 13:43:24 2024 +0300

Revert "[fix][sec] Upgrade Debezium oracle connector version to avoid… 
(#22668)
---
 pom.xml   | 1 -
 pulsar-io/debezium/oracle/pom.xml | 3 +--
 2 files changed, 1 insertion(+), 3 deletions(-)

diff --git a/pom.xml b/pom.xml
index bd231f92a4d..c817c8e1858 100644
--- a/pom.xml
+++ b/pom.xml
@@ -198,7 +198,6 @@ flexible messaging model and an intuitive client 
API.
 1.2.4
 8.12.1
 1.9.7.Final
-2.2.0.Final
 42.5.0
 8.0.30
 
diff --git a/pulsar-io/debezium/oracle/pom.xml 
b/pulsar-io/debezium/oracle/pom.xml
index 214e9c15c3a..1018d5f9573 100644
--- a/pulsar-io/debezium/oracle/pom.xml
+++ b/pulsar-io/debezium/oracle/pom.xml
@@ -48,8 +48,7 @@
 
   io.debezium
   debezium-connector-oracle
-  ${debezium.oracle.version}
-  runtime
+  ${debezium.version}

(pulsar) branch master updated: Revert "[fix][sec] Upgrade Debezium oracle connector version to avoid… (#22668)

This is an automated email from the ASF dual-hosted git repository.

technoboy pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new ca44b9bc7c4 Revert "[fix][sec] Upgrade Debezium oracle connector 
version to avoid… (#22668)
ca44b9bc7c4 is described below

commit ca44b9bc7c48eca59692744399872e1f14f4fe6f
Author: Lari Hotari 
AuthorDate: Wed May 8 13:43:24 2024 +0300

Revert "[fix][sec] Upgrade Debezium oracle connector version to avoid… 
(#22668)
---
 pom.xml   | 1 -
 pulsar-io/debezium/oracle/pom.xml | 3 +--
 2 files changed, 1 insertion(+), 3 deletions(-)

diff --git a/pom.xml b/pom.xml
index cec3b3c60db..c2f563eb60e 100644
--- a/pom.xml
+++ b/pom.xml
@@ -199,7 +199,6 @@ flexible messaging model and an intuitive client 
API.
 1.2.4
 8.12.1
 1.9.7.Final
-2.2.0.Final
 42.5.0
 8.0.30
 
diff --git a/pulsar-io/debezium/oracle/pom.xml 
b/pulsar-io/debezium/oracle/pom.xml
index b22a5785dfb..c69640ecff7 100644
--- a/pulsar-io/debezium/oracle/pom.xml
+++ b/pulsar-io/debezium/oracle/pom.xml
@@ -48,8 +48,7 @@
 
   io.debezium
   debezium-connector-oracle
-  ${debezium.oracle.version}
-  runtime
+  ${debezium.version}

Re: [PR] Revert "[fix][sec] Upgrade Debezium oracle connector version to avoid… [pulsar]



Technoboy- merged PR #22668:
URL: https://github.com/apache/pulsar/pull/22668


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(pulsar) branch branch-3.2 updated: [fix] Fix Reader can be stuck from transaction aborted messages. (#22610)

This is an automated email from the ASF dual-hosted git repository.

technoboy pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/branch-3.2 by this push:
 new 763f90f6dd3 [fix] Fix Reader can be stuck from transaction aborted 
messages. (#22610)
763f90f6dd3 is described below

commit 763f90f6dd317819d93990348bfc8519029c727d
Author: 道君 
AuthorDate: Tue May 7 20:45:16 2024 +0800

[fix] Fix Reader can be stuck from transaction aborted messages. (#22610)
---
 .../mledger/util/ManagedLedgerImplUtils.java   | 17 ++
 .../broker/service/persistent/PersistentTopic.java | 24 
 .../pulsar/broker/transaction/TransactionTest.java | 68 ++
 .../buffer/TopicTransactionBufferTest.java | 36 
 4 files changed, 110 insertions(+), 35 deletions(-)

diff --git 
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/util/ManagedLedgerImplUtils.java
 
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/util/ManagedLedgerImplUtils.java
index cd8671b0e62..01de115290a 100644
--- 
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/util/ManagedLedgerImplUtils.java
+++ 
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/util/ManagedLedgerImplUtils.java
@@ -38,11 +38,7 @@ public class ManagedLedgerImplUtils {
 final 
Predicate predicate,
 final 
PositionImpl startPosition) {
 CompletableFuture future = new CompletableFuture<>();
-if (!ledger.isValidPosition(startPosition)) {
-future.complete(startPosition);
-} else {
-internalAsyncReverseFindPositionOneByOne(ledger, predicate, 
startPosition, future);
-}
+internalAsyncReverseFindPositionOneByOne(ledger, predicate, 
startPosition, future);
 return future;
 }
 
@@ -50,6 +46,10 @@ public class ManagedLedgerImplUtils {
  final 
Predicate predicate,
  final 
PositionImpl position,
  final 
CompletableFuture future) {
+if (!ledger.isValidPosition(position)) {
+future.complete(position);
+return;
+}
 ledger.asyncReadEntry(position, new AsyncCallbacks.ReadEntryCallback() 
{
 @Override
 public void readEntryComplete(Entry entry, Object ctx) {
@@ -60,12 +60,7 @@ public class ManagedLedgerImplUtils {
 return;
 }
 PositionImpl previousPosition = 
ledger.getPreviousPosition((PositionImpl) position);
-if (!ledger.isValidPosition(previousPosition)) {
-future.complete(previousPosition);
-} else {
-internalAsyncReverseFindPositionOneByOne(ledger, 
predicate,
-ledger.getPreviousPosition((PositionImpl) 
position), future);
-}
+internalAsyncReverseFindPositionOneByOne(ledger, 
predicate, previousPosition, future);
 } catch (Exception e) {
 future.completeExceptionally(e);
 } finally {
diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
index 60eb700fc06..fa731b860f7 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
@@ -3561,18 +3561,18 @@ public class PersistentTopic extends AbstractTopic 
implements Topic, AddEntryCal
 
 @Override
 public CompletableFuture getLastDispatchablePosition() {
-PositionImpl maxReadPosition = getMaxReadPosition();
-// If `maxReadPosition` is not equal to `LastPosition`. It means that 
there are uncommitted transactions.
-// so return `maxRedPosition` directly.
-if (maxReadPosition.compareTo((PositionImpl) getLastPosition()) != 0) {
-return CompletableFuture.completedFuture(maxReadPosition);
-} else {
-return 
ManagedLedgerImplUtils.asyncGetLastValidPosition((ManagedLedgerImpl) ledger, 
entry -> {
-MessageMetadata md = 
Commands.parseMessageMetadata(entry.getDataBuffer());
-// If a messages has marker will filter by 
AbstractBaseDispatcher.filterEntriesForConsumer
-return !Markers.isServerOnlyMarker(md);
-}, maxReadPosition);
-}
+return

Re: [PR] [fix] [broker] Fix nothing changed after removing dynamic configs [pulsar]



Demogorgon314 commented on code in PR #22673:
URL: https://github.com/apache/pulsar/pull/22673#discussion_r1593814969


##
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java:
##
@@ -2488,40 +2488,71 @@ private void handleDynamicConfigurationUpdates() {
 
 if (dynamicConfigResources != null) {
 dynamicConfigResources.getDynamicConfigurationAsync()
-.thenAccept(optMap -> {
-if (!optMap.isPresent()) {
-return;
+.thenAccept(optMap -> {
+// Case some dynamic configs have been removed.
+dynamicConfigurationMap.forEach((configKey, fieldWrapper) 
-> {
+boolean configRemoved = optMap.isEmpty() || 
!optMap.get().containsKey(configKey);
+if (fieldWrapper.lastDynamicValue != null && 
configRemoved) {
+configValueChanged(configKey, null);
 }
-Map data = optMap.get();
-data.forEach((configKey, value) -> {
-ConfigField configFieldWrapper = 
dynamicConfigurationMap.get(configKey);
-if (configFieldWrapper == null) {
-log.warn("{} does not exist in 
dynamicConfigurationMap, skip this config.", configKey);
-return;
-}
-Field configField = configFieldWrapper.field;
-Consumer listener = 
configRegisteredListeners.get(configKey);
-try {
-final Object existingValue;
-final Object newValue;
-if (configField != null) {
-newValue = 
FieldParser.value(data.get(configKey), configField);
-existingValue = 
configField.get(pulsar.getConfiguration());
-configField.set(pulsar.getConfiguration(), 
newValue);
-} else {
-newValue = value;
-existingValue = 
configFieldWrapper.customValue;
-configFieldWrapper.customValue = newValue 
== null ? null : String.valueOf(newValue);
-}
-log.info("Successfully updated configuration 
{}/{}", configKey, data.get(configKey));
-if (listener != null && 
!Objects.equals(existingValue, newValue)) {
-listener.accept(newValue);
-}
-} catch (Exception e) {
-log.error("Failed to update config {}", 
configKey, e);
-}
-});
 });
+// Some configs have been changed.
+if (!optMap.isPresent()) {
+return;
+}
+Map data = optMap.get();
+data.forEach((configKey, value) -> {
+configValueChanged(configKey, value);
+});
+});
+}
+}
+
+private void configValueChanged(String configKey, String newValueStr) {
+ConfigField configFieldWrapper = 
dynamicConfigurationMap.get(configKey);
+if (configFieldWrapper == null) {
+log.warn("{} does not exist in dynamicConfigurationMap, skip this 
config.", configKey);
+return;
+}
+Consumer listener = configRegisteredListeners.get(configKey);
+try {
+// Convert existingValue and newValue.
+final Object existingValue;
+final Object newValue;
+if (configFieldWrapper.field != null) {
+if (StringUtils.isBlank(newValueStr)) {
+newValue = configFieldWrapper.defaultValue;
+} else {
+newValue = FieldParser.value(newValueStr, 
configFieldWrapper.field);
+}
+existingValue = 
configFieldWrapper.field.get(pulsar.getConfiguration());
+configFieldWrapper.field.set(pulsar.getConfiguration(), 
newValue);
+} else {
+// This case only occurs when it is a customized item.
+// See: 
https://github.com/apache/pulsar/blob/master/pip/pip-300.md.
+log.info("k:{} v:{} Skip to update configuration in memory 
because it is a customized item, only"
++ " trigger an event listeners.", configKey, 
newValueStr);

Review Comment:
   ```suggestion
   log.info("Skip update customized dynamic configuration {}/{} 
in

Re: [PR] [improve] [pip] PIP-349: Add additionalSystemCursorNames ignore list for TTL check [pulsar]



RobertIndie commented on code in PR #22651:
URL: https://github.com/apache/pulsar/pull/22651#discussion_r1593816786


##
pip/pip-349.md:
##
@@ -0,0 +1,33 @@
+# PIP-349: Add additionalSystemCursorNames ignore list for ttl check
+
+# Background knowledge
+
+In Pulsar topic, we have [retention 
policy](https://pulsar.apache.org/docs/3.2.x/cookbooks-retention-expiry/#retention-policies)
 to control the acknowledged message lifetime. For the unacknowledged messages, 
we have a separate mechanism to control the message lifetime, which is called 
[`TTL`](https://pulsar.apache.org/docs/3.2.x/cookbooks-retention-expiry/#time-to-live-ttl).
 The `TTL` is a time-to-live value for the message, which is controlled by 
`ttlDurationDefaultInSeconds`. The message will be automatically acknowledged 
if it is not consumed within the `TTL` value.
+
+# Motivation
+
+In Pulsar, we have two kinds of topics, system topic and normal topic. The 
system topics are used for internal purposes, such as transaction internal 
topics. The system topics are not supposed to be consumed by the users. 
However, the system topics are still subject to the `TTL` check. If the system 
topics are not consumed within the `TTL` value, the messages in the system 
topics will be automatically acknowledged. This is not the expected behavior 
for the system topics and may lead to data loss.
+For normal topics, we also has two kinds of subscriptions, system subscription 
and normal subscription. The system subscription is used for internal purposes, 
such as compaction service or third-party plugins. The system subscription is 
not supposed to be used by the users. However, the system subscription is still 
subject to the `TTL` check. If the system subscription is not consumed within 
the `TTL` value, the messages in the system subscription will be automatically 
acknowledged. This is not the expected behavior for the system subscription.
+
+We had one PR [#21865](https://github.com/apache/pulsar/pull/21865) to filter 
the compaction service cursors for TTL check, but it doesn't cover other system 
cursors. To provide a general solution and support third-party plugin cursors 
not impacted by TTL, I proposed to add an additionalSystemCursorNames ignore 
list to filter the TTL check.
+
+# Goals
+
+## In Scope
+
+Add an additionalSystemCursorNames ignore list to filter the TTL check for 
additional system subscriptions except for compaction service subscription. The 
systemCursorNames ignore list is an optional configuration, and the default 
value is empty. Pulsar broker will filter the TTL check for the 
additionalSystemCursorNames subscriptions.
+The compaction service subscription is a system subscription and should not be 
impacted by TTL. To reduce the risk of data loss after enabled compaction 
service, we will add the compaction service subscription to the TTL ignore list 
by default and can't be removed.
+
+# Detailed Design
+
+## Design & Implementation Details
+
+Add a systemCursorNames ignore list to filter the TTL check for system 
subscriptions. The systemCursorNames ignore list is an optional configuration, 
and the default value is empty. Pulsar broker will filter the TTL check for the 
additionalSystemCursorNames subscriptions.

Review Comment:
   ```suggestion
   Add a additionalSystemCursorNames ignore list to filter the TTL check for 
system subscriptions. The additionalSystemCursorNames ignore list is an 
optional configuration, and the default value is empty. Pulsar broker will 
filter the TTL check for the additionalSystemCursorNames subscriptions.
   ```



##
pip/pip-349.md:
##
@@ -0,0 +1,33 @@
+# PIP-349: Add additionalSystemCursorNames ignore list for ttl check
+
+# Background knowledge
+
+In Pulsar topic, we have [retention 
policy](https://pulsar.apache.org/docs/3.2.x/cookbooks-retention-expiry/#retention-policies)
 to control the acknowledged message lifetime. For the unacknowledged messages, 
we have a separate mechanism to control the message lifetime, which is called 
[`TTL`](https://pulsar.apache.org/docs/3.2.x/cookbooks-retention-expiry/#time-to-live-ttl).
 The `TTL` is a time-to-live value for the message, which is controlled by 
`ttlDurationDefaultInSeconds`. The message will be automatically acknowledged 
if it is not consumed within the `TTL` value.
+
+# Motivation
+
+In Pulsar, we have two kinds of topics, system topic and normal topic. The 
system topics are used for internal purposes, such as transaction internal 
topics. The system topics are not supposed to be consumed by the users. 
However, the system topics are still subject to the `TTL` check. If the system 
topics are not consumed within the `TTL` value, the messages in the system 
topics will be automatically acknowledged. This is not the expected behavior 
for the system topics and may lead to data loss.
+For normal topics, we also has two kinds of subscriptions, system subscription 
and normal subscription. The system

(pulsar) branch branch-3.2 updated: [fix][broker] avoid offload system topic (#22497)

This is an automated email from the ASF dual-hosted git repository.

technoboy pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/branch-3.2 by this push:
 new c3c17dee756 [fix][broker] avoid offload system topic (#22497)
c3c17dee756 is described below

commit c3c17dee7567d0a182affb1991e1e35098689d9b
Author: Qiang Zhao 
AuthorDate: Wed May 8 13:10:49 2024 +0800

[fix][broker] avoid offload system topic (#22497)

Co-authored-by: 道君 
---
 .../pulsar/broker/service/BrokerService.java   |  8 +-
 .../pulsar/broker/service/BrokerServiceTest.java   | 94 ++
 2 files changed, 101 insertions(+), 1 deletion(-)

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
index 032d4dd9369..60d56c0d908 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
@@ -1906,7 +1906,13 @@ public class BrokerService implements Closeable {
 topicLevelOffloadPolicies,
 
OffloadPoliciesImpl.oldPoliciesCompatible(nsLevelOffloadPolicies, 
policies.orElse(null)),
 getPulsar().getConfig().getProperties());
-if 
(NamespaceService.isSystemServiceNamespace(namespace.toString())) {
+if (NamespaceService.isSystemServiceNamespace(namespace.toString())
+|| SystemTopicNames.isSystemTopic(topicName)) {
+/*
+ Avoid setting broker internal system topics using off-loader 
because some of them are the
+ preconditions of other topics. The slow replying log speed 
will cause a delay in all the topic
+ loading.(timeout)
+ */
 
managedLedgerConfig.setLedgerOffloader(NullLedgerOffloader.INSTANCE);
 } else  {
 if (topicLevelOffloadPolicies != null) {
diff --git 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BrokerServiceTest.java
 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BrokerServiceTest.java
index fcf11fad708..ab0b8f813ea 100644
--- 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BrokerServiceTest.java
+++ 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BrokerServiceTest.java
@@ -67,12 +67,15 @@ import java.util.concurrent.atomic.AtomicBoolean;
 import java.util.concurrent.atomic.AtomicReference;
 import lombok.Cleanup;
 import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.client.api.ReadHandle;
+import org.apache.bookkeeper.mledger.LedgerOffloader;
 import org.apache.bookkeeper.mledger.ManagedLedgerConfig;
 import org.apache.bookkeeper.mledger.ManagedLedgerException;
 import org.apache.bookkeeper.mledger.impl.ManagedCursorImpl;
 import org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl;
 import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
 import org.apache.commons.lang3.StringUtils;
+import org.apache.bookkeeper.mledger.impl.NullLedgerOffloader;
 import org.apache.http.HttpResponse;
 import org.apache.http.client.methods.HttpGet;
 import org.apache.http.impl.client.CloseableHttpClient;
@@ -111,6 +114,9 @@ import org.apache.pulsar.common.naming.SystemTopicNames;
 import org.apache.pulsar.common.naming.TopicName;
 import org.apache.pulsar.common.policies.data.BundlesData;
 import org.apache.pulsar.common.policies.data.LocalPolicies;
+import org.apache.pulsar.common.policies.data.OffloadPolicies;
+import org.apache.pulsar.common.policies.data.OffloadPoliciesImpl;
+import org.apache.pulsar.common.policies.data.OffloadedReadPriority;
 import org.apache.pulsar.common.policies.data.SubscriptionStats;
 import org.apache.pulsar.common.policies.data.TopicStats;
 import org.apache.pulsar.common.protocol.Commands;
@@ -1772,4 +1778,92 @@ public class BrokerServiceTest extends BrokerTestBase {
 fail("Unsubscribe failed");
 }
 }
+
+
+@Test
+public void testOffloadConfShouldNotAppliedForSystemTopic() throws 
PulsarAdminException {
+final String driver = "aws-s3";
+final String region = "test-region";
+final String bucket = "test-bucket";
+final String role = "test-role";
+final String roleSessionName = "test-role-session-name";
+final String credentialId = "test-credential-id";
+final String credentialSecret = "test-credential-secret";
+final String endPoint = "test-endpoint";
+final Integer maxBlockSizeInBytes = 5;
+final Integer readBufferSizeInBytes = 2;
+final Long offloadThresholdInBytes = 10L;
+final Long offloadThresholdInSeconds = 1000L;
+final Long offloadDeletionLagInMillis = 5L;
+
+final OffloadPoliciesImpl

[PR] [Fix][broker] Limit replication rate based on bytes [pulsar]



nodece opened a new pull request, #22674:
URL: https://github.com/apache/pulsar/pull/22674

   ### Motivation
   
   When the replicator rate limit value based on bytes is less than the 
`org.apache.pulsar.broker.service.persistent.PersistentReplicator#readMaxSizeBytes`,
 this will cause the outgoing bytes to exceed the limit value.
   
   The goal is to make outgoing bytes as close to the rate limit value as 
possible.
   
   ### Modifications
   
   - 
`org.apache.pulsar.broker.service.persistent.PersistentReplicator#getAvailablePermits`
 returns the available messages and bytes, and then reads the entries based on 
that.
   
   ### Verifying this change
   
   The test has been added.
   
   ### Documentation
   
   
   
   - [ ] `doc` 
   - [ ] `doc-required` 
   - [x] `doc-not-needed` 
   - [ ] `doc-complete` 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [fix] [broker] Fix nothing changed after removing dynamic configs [pulsar]



Demogorgon314 commented on code in PR #22673:
URL: https://github.com/apache/pulsar/pull/22673#discussion_r1593814969


##
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java:
##
@@ -2488,40 +2488,71 @@ private void handleDynamicConfigurationUpdates() {
 
 if (dynamicConfigResources != null) {
 dynamicConfigResources.getDynamicConfigurationAsync()
-.thenAccept(optMap -> {
-if (!optMap.isPresent()) {
-return;
+.thenAccept(optMap -> {
+// Case some dynamic configs have been removed.
+dynamicConfigurationMap.forEach((configKey, fieldWrapper) 
-> {
+boolean configRemoved = optMap.isEmpty() || 
!optMap.get().containsKey(configKey);
+if (fieldWrapper.lastDynamicValue != null && 
configRemoved) {
+configValueChanged(configKey, null);
 }
-Map data = optMap.get();
-data.forEach((configKey, value) -> {
-ConfigField configFieldWrapper = 
dynamicConfigurationMap.get(configKey);
-if (configFieldWrapper == null) {
-log.warn("{} does not exist in 
dynamicConfigurationMap, skip this config.", configKey);
-return;
-}
-Field configField = configFieldWrapper.field;
-Consumer listener = 
configRegisteredListeners.get(configKey);
-try {
-final Object existingValue;
-final Object newValue;
-if (configField != null) {
-newValue = 
FieldParser.value(data.get(configKey), configField);
-existingValue = 
configField.get(pulsar.getConfiguration());
-configField.set(pulsar.getConfiguration(), 
newValue);
-} else {
-newValue = value;
-existingValue = 
configFieldWrapper.customValue;
-configFieldWrapper.customValue = newValue 
== null ? null : String.valueOf(newValue);
-}
-log.info("Successfully updated configuration 
{}/{}", configKey, data.get(configKey));
-if (listener != null && 
!Objects.equals(existingValue, newValue)) {
-listener.accept(newValue);
-}
-} catch (Exception e) {
-log.error("Failed to update config {}", 
configKey, e);
-}
-});
 });
+// Some configs have been changed.
+if (!optMap.isPresent()) {
+return;
+}
+Map data = optMap.get();
+data.forEach((configKey, value) -> {
+configValueChanged(configKey, value);
+});
+});
+}
+}
+
+private void configValueChanged(String configKey, String newValueStr) {
+ConfigField configFieldWrapper = 
dynamicConfigurationMap.get(configKey);
+if (configFieldWrapper == null) {
+log.warn("{} does not exist in dynamicConfigurationMap, skip this 
config.", configKey);
+return;
+}
+Consumer listener = configRegisteredListeners.get(configKey);
+try {
+// Convert existingValue and newValue.
+final Object existingValue;
+final Object newValue;
+if (configFieldWrapper.field != null) {
+if (StringUtils.isBlank(newValueStr)) {
+newValue = configFieldWrapper.defaultValue;
+} else {
+newValue = FieldParser.value(newValueStr, 
configFieldWrapper.field);
+}
+existingValue = 
configFieldWrapper.field.get(pulsar.getConfiguration());
+configFieldWrapper.field.set(pulsar.getConfiguration(), 
newValue);
+} else {
+// This case only occurs when it is a customized item.
+// See: 
https://github.com/apache/pulsar/blob/master/pip/pip-300.md.
+log.info("k:{} v:{} Skip to update configuration in memory 
because it is a customized item, only"
++ " trigger an event listeners.", configKey, 
newValueStr);

Review Comment:
   ```suggestion
   log.info("Skip to update configuration {}/{} in memory 
because

(pulsar) branch branch-3.2 updated: [improve][ws] Add memory limit configuration for Pulsar client used in Websocket proxy (#22666)

This is an automated email from the ASF dual-hosted git repository.

technoboy pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/branch-3.2 by this push:
 new e32cdfb113b [improve][ws] Add memory limit configuration for Pulsar 
client used in Websocket proxy (#22666)
e32cdfb113b is described below

commit e32cdfb113b1693a2420e39ab40f985b59a44899
Author: Lari Hotari 
AuthorDate: Wed May 8 06:56:35 2024 +0300

[improve][ws] Add memory limit configuration for Pulsar client used in 
Websocket proxy (#22666)
---
 conf/broker.conf   | 3 +++
 conf/standalone.conf   | 3 +++
 conf/websocket.conf| 3 +++
 .../main/java/org/apache/pulsar/broker/ServiceConfiguration.java   | 7 +++
 .../main/java/org/apache/pulsar/websocket/WebSocketService.java| 3 ++-
 .../pulsar/websocket/service/WebSocketProxyConfiguration.java  | 3 +++
 6 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/conf/broker.conf b/conf/broker.conf
index dd0f3e49e1f..e0ebbe3043a 100644
--- a/conf/broker.conf
+++ b/conf/broker.conf
@@ -1539,6 +1539,9 @@ webSocketNumServiceThreads=
 # Number of connections per Broker in Pulsar Client used in WebSocket proxy
 webSocketConnectionsPerBroker=
 
+# Memory limit in MBs for direct memory in Pulsar Client used in WebSocket 
proxy
+webSocketPulsarClientMemoryLimitInMB=0
+
 # Time in milliseconds that idle WebSocket session times out
 webSocketSessionIdleTimeoutMillis=30
 
diff --git a/conf/standalone.conf b/conf/standalone.conf
index 316143ab49d..5eb9fadcf19 100644
--- a/conf/standalone.conf
+++ b/conf/standalone.conf
@@ -967,6 +967,9 @@ webSocketNumIoThreads=8
 # Number of connections per Broker in Pulsar Client used in WebSocket proxy
 webSocketConnectionsPerBroker=8
 
+# Memory limit in MBs for direct memory in Pulsar Client used in WebSocket 
proxy
+webSocketPulsarClientMemoryLimitInMB=0
+
 # Time in milliseconds that idle WebSocket session times out
 webSocketSessionIdleTimeoutMillis=30
 
diff --git a/conf/websocket.conf b/conf/websocket.conf
index 9051f3b590c..91f7f7d4c23 100644
--- a/conf/websocket.conf
+++ b/conf/websocket.conf
@@ -71,6 +71,9 @@ numHttpServerThreads=
 # Number of connections per Broker in Pulsar Client used in WebSocket proxy
 webSocketConnectionsPerBroker=
 
+# Memory limit in MBs for direct memory in Pulsar Client used in WebSocket 
proxy
+webSocketPulsarClientMemoryLimitInMB=0
+
 # Time in milliseconds that idle WebSocket session times out
 webSocketSessionIdleTimeoutMillis=30
 
diff --git 
a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
 
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
index d1f2e9b585f..f53eb7e183f 100644
--- 
a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
+++ 
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
@@ -2892,6 +2892,13 @@ public class ServiceConfiguration implements 
PulsarConfiguration {
 doc = "Number of connections per Broker in Pulsar Client used in 
WebSocket proxy"
 )
 private int webSocketConnectionsPerBroker = 
Runtime.getRuntime().availableProcessors();
+
+@FieldContext(
+category = CATEGORY_WEBSOCKET,
+doc = "Memory limit in MBs for direct memory in Pulsar Client used 
in WebSocket proxy"
+)
+private int webSocketPulsarClientMemoryLimitInMB = 0;
+
 @FieldContext(
 category = CATEGORY_WEBSOCKET,
 doc = "Time in milliseconds that idle WebSocket session times out"
diff --git 
a/pulsar-websocket/src/main/java/org/apache/pulsar/websocket/WebSocketService.java
 
b/pulsar-websocket/src/main/java/org/apache/pulsar/websocket/WebSocketService.java
index 66b2a0075ec..889f4431cc3 100644
--- 
a/pulsar-websocket/src/main/java/org/apache/pulsar/websocket/WebSocketService.java
+++ 
b/pulsar-websocket/src/main/java/org/apache/pulsar/websocket/WebSocketService.java
@@ -195,7 +195,8 @@ public class WebSocketService implements Closeable {
 
 private PulsarClient createClientInstance(ClusterData clusterData) throws 
IOException {
 ClientBuilder clientBuilder = PulsarClient.builder() //
-.memoryLimit(0, SizeUnit.BYTES)
+
.memoryLimit(SizeUnit.MEGA_BYTES.toBytes(config.getWebSocketPulsarClientMemoryLimitInMB()),
+SizeUnit.BYTES)
 .statsInterval(0, TimeUnit.SECONDS) //
 .enableTls(config.isTlsEnabled()) //
 
.allowTlsInsecureConnection(config.isTlsAllowInsecureConnection()) //
diff --git 
a/pulsar-websocket/src/main/java/org/apache/pulsar/websocket/service/WebSocketProxyConfiguration.java

[I] Pulsar manager service fails saying "2024-05-07 15:02:17,476 INFO exited: pulsar-manager-backend (terminated by SIGKILL; not expected)" [pulsar-helm-chart]



bhavyaravilla opened a new issue, #496:
URL: https://github.com/apache/pulsar-helm-chart/issues/496

   **Describe the bug**
   I am using the latest pulsar helm chart to deploy pulsar. We had a 
requirement to add the pulsar-manager. Enabled pulsar manager. I've provided 
database url, username and password. We are using a postgres DB (AWS RDS). The 
pod comes up. But the service keeps failing with the below message.
   
   ```
   2024-05-07 15:01:47,078 INFO spawned: 'pulsar-manager-backend' with pid 6805
   2024-05-07 15:01:48,955 INFO success: pulsar-manager-backend entered RUNNING 
state, process has stayed up for > than 1 seconds (startsecs)
   2024-05-07 15:02:17,476 INFO exited: pulsar-manager-backend (terminated by 
SIGKILL; not expected)
   ```
   
   There are no other logs to see whats wrong. The pulsar-manager logs dont 
show anything as well. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [fix] [broker] Fix nothing changed after removing dynamic configs [pulsar]



poorbarcode commented on PR #22673:
URL: https://github.com/apache/pulsar/pull/22673#issuecomment-2100239097

   /pulsarbot rerun-failure-checks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [fix] [broker] rename to changeMaxReadPositionCount [pulsar]



codecov-commenter commented on PR #22656:
URL: https://github.com/apache/pulsar/pull/22656#issuecomment-2100231003

   ## 
[Codecov](https://app.codecov.io/gh/apache/pulsar/pull/22656?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
 Report
   All modified and coverable lines are covered by tests :white_check_mark:
   > Project coverage is 73.12%. Comparing base 
[(`bbc6224`)](https://app.codecov.io/gh/apache/pulsar/commit/bbc62245c5ddba1de4b1e7cee4ab49334bc36277?dropdown=coverage=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
 to head 
[(`3e2e3a4`)](https://app.codecov.io/gh/apache/pulsar/pull/22656?dropdown=coverage=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache).
   > Report is 240 commits behind head on master.
   
   Additional details and impacted files
   
   
   [![Impacted file tree 
graph](https://app.codecov.io/gh/apache/pulsar/pull/22656/graphs/tree.svg?width=650=150=pr=acYqCpsK9J_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)](https://app.codecov.io/gh/apache/pulsar/pull/22656?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
   
   ```diff
   @@ Coverage Diff  @@
   ## master   #22656  +/-   ##
   
   - Coverage 73.57%   73.12%   -0.45% 
   + Complexity32624 2401   -30223 
   
 Files  1877 1887  +10 
 Lines139502   141068+1566 
 Branches  1529915482 +183 
   
   + Hits 102638   103163 +525 
   - Misses2890829925+1017 
   - Partials   7956 7980  +24 
   ```
   
   | 
[Flag](https://app.codecov.io/gh/apache/pulsar/pull/22656/flags?src=pr=flags_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
 | Coverage Δ | |
   |---|---|---|
   | 
[inttests](https://app.codecov.io/gh/apache/pulsar/pull/22656/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
 | `27.33% <71.42%> (+2.74%)` | :arrow_up: |
   | 
[systests](https://app.codecov.io/gh/apache/pulsar/pull/22656/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
 | `24.61% <0.00%> (+0.28%)` | :arrow_up: |
   | 
[unittests](https://app.codecov.io/gh/apache/pulsar/pull/22656/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
 | `72.15% <100.00%> (-0.70%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click 
here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#carryforward-flags-in-the-pull-request-comment)
 to find out more.
   
   | 
[Files](https://app.codecov.io/gh/apache/pulsar/pull/22656?dropdown=coverage=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
 | Coverage Δ | |
   |---|---|---|
   | 
[...ransaction/buffer/impl/TopicTransactionBuffer.java](https://app.codecov.io/gh/apache/pulsar/pull/22656?src=pr=tree=pulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Ftransaction%2Fbuffer%2Fimpl%2FTopicTransactionBuffer.java_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci90cmFuc2FjdGlvbi9idWZmZXIvaW1wbC9Ub3BpY1RyYW5zYWN0aW9uQnVmZmVyLmphdmE=)
 | `88.16% <100.00%> (+0.41%)` | :arrow_up: |
   
   ... and [337 files with indirect coverage 
changes](https://app.codecov.io/gh/apache/pulsar/pull/22656/indirect-changes?src=pr=tree-more_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [improve][broker] Remove unused method CompactionRecord.reset [pulsar]



codecov-commenter commented on PR #22670:
URL: https://github.com/apache/pulsar/pull/22670#issuecomment-2100207957

   ## 
[Codecov](https://app.codecov.io/gh/apache/pulsar/pull/22670?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
 Report
   All modified and coverable lines are covered by tests :white_check_mark:
   > Project coverage is 72.73%. Comparing base 
[(`bbc6224`)](https://app.codecov.io/gh/apache/pulsar/commit/bbc62245c5ddba1de4b1e7cee4ab49334bc36277?dropdown=coverage=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
 to head 
[(`479bb70`)](https://app.codecov.io/gh/apache/pulsar/pull/22670?dropdown=coverage=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache).
   > Report is 240 commits behind head on master.
   
   Additional details and impacted files
   
   
   [![Impacted file tree 
graph](https://app.codecov.io/gh/apache/pulsar/pull/22670/graphs/tree.svg?width=650=150=pr=acYqCpsK9J_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)](https://app.codecov.io/gh/apache/pulsar/pull/22670?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
   
   ```diff
   @@ Coverage Diff  @@
   ## master   #22670  +/-   ##
   
   - Coverage 73.57%   72.73%   -0.85% 
   + Complexity3262432550  -74 
   
 Files  1877 1887  +10 
 Lines139502   141053+1551 
 Branches  1529915481 +182 
   
   - Hits 102638   102590  -48 
   - Misses2890830584+1676 
   + Partials   7956 7879  -77 
   ```
   
   | 
[Flag](https://app.codecov.io/gh/apache/pulsar/pull/22670/flags?src=pr=flags_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
 | Coverage Δ | |
   |---|---|---|
   | 
[inttests](https://app.codecov.io/gh/apache/pulsar/pull/22670/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
 | `27.43% <ø> (+2.85%)` | :arrow_up: |
   | 
[systests](https://app.codecov.io/gh/apache/pulsar/pull/22670/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
 | `24.70% <ø> (+0.38%)` | :arrow_up: |
   | 
[unittests](https://app.codecov.io/gh/apache/pulsar/pull/22670/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
 | `71.49% <ø> (-1.35%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click 
here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#carryforward-flags-in-the-pull-request-comment)
 to find out more.
   
   | 
[Files](https://app.codecov.io/gh/apache/pulsar/pull/22670?dropdown=coverage=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
 | Coverage Δ | |
   |---|---|---|
   | 
[...org/apache/pulsar/compaction/CompactionRecord.java](https://app.codecov.io/gh/apache/pulsar/pull/22670?src=pr=tree=pulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fcompaction%2FCompactionRecord.java_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NvbXBhY3Rpb24vQ29tcGFjdGlvblJlY29yZC5qYXZh)
 | `100.00% <ø> (ø)` | |
   | 
[.../apache/pulsar/compaction/CompactorMXBeanImpl.java](https://app.codecov.io/gh/apache/pulsar/pull/22670?src=pr=tree=pulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fcompaction%2FCompactorMXBeanImpl.java_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NvbXBhY3Rpb24vQ29tcGFjdG9yTVhCZWFuSW1wbC5qYXZh)
 | `94.44% <ø> (-0.56%)` | :arrow_down: |
   
   ... and [353 files with indirect coverage 
changes](https://app.codecov.io/gh/apache/pulsar/pull/22670/indirect-changes?src=pr=tree-more_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [fix] [broker] rename to changeMaxReadPositionCount [pulsar]



thetumbled commented on PR #22656:
URL: https://github.com/apache/pulsar/pull/22656#issuecomment-2100176006

   /pulsarbot rerun-failure-checks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[I] 不使用证书的情况下调用ssl的url，服务无法连接 [pulsar-client-node]



shuaiqiha opened a new issue, #382:
URL: https://github.com/apache/pulsar-client-node/issues/382

   以下是node代码，在java中是可以通过```
   PulsarClient client = 
PulsarClient.builder().serviceUrl(brokerServerUrl).allowTlsInsecureConnection(true)
   .authentication(new IoTAuthentication(iotAccessId, 
iotSecretKey)).build();
   Consumer consumer = 
client.newConsumer(Schema.STRING).topic(String.format("%s/iot/event", 
iotAccessId))
   
.subscriptionName(subscriptionName).subscriptionType(SubscriptionType.Failover)
   .autoUpdatePartitions(Boolean.FALSE).subscribe();
   ```
   这种方式来跳过ssl证书的，在下面node中我也加入了tlsAllowInsecureConnection: true,
   tlsValidateHostname: false,这两个参数但是无法连接到服务
   
   
   
   `const Pulsar = require('pulsar-client');
   const crypto = require('crypto');
   const iotAccessId = "yGmZyJnpqPPIGwzj1564";
   const iotSecretKey="b9fab8817fa64f4ab6424d6a7035d2e7";
   const brokerSSLServerUrl = "pulsar+ssl://iot-north-mq.heclouds.com:6651/";
   const iotSubscriptionName="yGmZyJnpqPPIGwzj1564-sub";
   
   const token = JSON.stringify({
   tenant: iotAccessId,
   password: sha256(iotAccessId + sha256(iotSecretKey)).substring(4, 20)
   });
   
   (async () => {
   try {
   console.log("token:", token);
   // Create a client
   let authenticationToken = new Pulsar.AuthenticationToken({
   token: token
   });
   
   Pulsar.Client.setLogHandler((level, file, line, message) => {
   console.log('[%s][%s:%d] %s', Pulsar.LogLevel.toString(level), 
file, line, message);
   });
   
   const client = new Pulsar.Client({
   serviceUrl: brokerSSLServerUrl,
   authentication: authenticationToken,
   tlsAllowInsecureConnection: true,
   tlsValidateHostname: false,
   });
   
   
   console.log("client:", client);
   
   // Create a consumer
   const consumer = await client.subscribe({
   topic: `${iotAccessId}/iot/event`,
   // topic: 
'persistent://yGmZyJnpqPPIGwzj1564/iot/event-partition-0',
   subscription: iotSubscriptionName,
   subscriptionType: 'Failover'
   });
   
   console.log("consumer:", consumer);
   
   // Receive the message
   const msg = await consumer.receive();
   console.log(msg.getData().toString());
   consumer.acknowledge(msg);
   
   await consumer.close();
   await client.close();
   } catch (error) {
   console.error("An error occurred:", error.message);
   console.error(error.stack);
   }
   })();
   
   function sha256(data) {
   return crypto.createHash('sha256').update(data, 'utf8').digest('hex');
   }`
   
   
   最下面是报错信息```
   [INFO][Client:86] Subscribing on Topic :yGmZyJnpqPPIGwzj1564/iot/event
   [INFO][ClientConnection:187] [ -> 
pulsar+ssl://iot-north-mq.heclouds.com:6651] Create ClientConnection, 
timeout=1
   [INFO][ConnectionPool:124] Created connection for 
pulsar+ssl://iot-north-mq.heclouds.com:6651-pulsar+ssl://iot-north-mq.heclouds.com:6651-0
   [INFO][ClientConnection:403] [10.10.80.213:63773 -> 218.201.45.7:6651] 
Connected to broker
   [ERROR][ClientConnection:1318] [10.10.80.213:63773 -> 218.201.45.7:6651] 
Connection closed with ConnectError (refCnt: 2)
   [INFO][ConnectionPool:141] Remove connection for 
pulsar+ssl://iot-north-mq.heclouds.com:6651-pulsar+ssl://iot-north-mq.heclouds.com:6651-0
   [ERROR][ClientImpl:507] Error Checking/Getting Partition Metadata while 
Subscribing on persistent://yGmZyJnpqPPIGwzj1564/iot/event -- ConnectError
   [INFO][ClientConnection:275] [10.10.80.213:63773 -> 218.201.45.7:6651] 
Destroyed connection to pulsar+ssl://iot-north-mq.heclouds.com:6651-0
   An error occurred: Failed to create consumer: ConnectError
   Error: Failed to create consumer: ConnectError
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] [fix] [broker] Fix nothing happend after remove a dynamic config [pulsar]



poorbarcode opened a new pull request, #22673:
URL: https://github.com/apache/pulsar/pull/22673

   ### Motivation
   
   **Background**
   - Broker registers listeners to monitor the config change events
   - After calling `admin updateDynamicConfiguration`, the configuration in 
memory will be changed and the listeners will receive an event.
   
   **Issue**
   After calling `admin deleteDynamicConfiguration`, the configuration in 
memory will not change and the listeners will receive nothing.
   
   ### Modifications
   
   When calling `admin deleteDynamicConfiguration`
   - revert the config to the default value
   - trigger an event for the listeners
   
   ### Documentation
   
   
   
   - [ ] `doc` 
   - [ ] `doc-required` 
   - [x] `doc-not-needed` 
   - [ ] `doc-complete` 
   
   ### Matching PR in forked repository
   
   PR in forked repository: x
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [fix][offload] Support offload prefetchrounds & assignmentThreads adjustable [pulsar]



tisonkun closed pull request #21034: [fix][offload] Support offload 
prefetchrounds & assignmentThreads adjustable
URL: https://github.com/apache/pulsar/pull/21034


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [fix][offload] Support offload prefetchrounds & assignmentThreads adjustable [pulsar]



tisonkun commented on PR #21034:
URL: https://github.com/apache/pulsar/pull/21034#issuecomment-2100129529

   Closed as stale and conflict. Sorry we don't pick up this PR in time :(


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [feat] Enable hostname verification by default [pulsar]



tisonkun commented on PR #20268:
URL: https://github.com/apache/pulsar/pull/20268#issuecomment-2100126134

   Closed as stale and conflict.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [feat] Enable hostname verification by default [pulsar]



tisonkun closed pull request #20268: [feat] Enable hostname verification by 
default
URL: https://github.com/apache/pulsar/pull/20268


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [improve][broker, client] Fixing BC FIPS based payload encryption [pulsar]



tisonkun commented on PR #20542:
URL: https://github.com/apache/pulsar/pull/20542#issuecomment-2100120153

   Closed as stale and conlfict.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [improve][broker, client] Fixing BC FIPS based payload encryption [pulsar]



tisonkun closed pull request #20542: [improve][broker, client] Fixing BC FIPS 
based payload encryption
URL: https://github.com/apache/pulsar/pull/20542


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [fix][broker] usedLocallySinceLastReport should always be reset [pulsar]



nodece closed pull request #22672: [fix][broker] usedLocallySinceLastReport 
should always be reset
URL: https://github.com/apache/pulsar/pull/22672


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [improve][io]add metrics to elastic search sink [pulsar]



tisonkun commented on PR #20498:
URL: https://github.com/apache/pulsar/pull/20498#issuecomment-2100119505

   Closed as stale and conflict.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [improve][io]add metrics to elastic search sink [pulsar]



tisonkun closed pull request #20498: [improve][io]add metrics to elastic search 
sink
URL: https://github.com/apache/pulsar/pull/20498


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [improve][pip] PIP-347: add role field in consumer's stat [pulsar]



thetumbled commented on PR #22564:
URL: https://github.com/apache/pulsar/pull/22564#issuecomment-2100119625

   WDYT, should we expose it to regular users(not everyone actually, but those 
who own the lookup permission of this topic)? @eolivelli @lhotari 
@codelipenghui @Technoboy- @michaeljmarshall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [fix][broker] Introduce the last sent position to fix message ordering issues in Key_Shared (PIP-282) [pulsar]



equanz commented on code in PR #21953:
URL: https://github.com/apache/pulsar/pull/21953#discussion_r1593651866


##
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java:
##
@@ -312,6 +365,62 @@ protected synchronized boolean 
trySendMessagesToConsumers(ReadType readType, Lis
 }
 }
 
+// Update the last sent position and remove ranges from 
individuallySentPositions if necessary
+if (!allowOutOfOrderDelivery && lastSentPosition != null) {
+final ManagedLedgerImpl managedLedger = ((ManagedLedgerImpl) 
cursor.getManagedLedger());
+com.google.common.collect.Range range = 
individuallySentPositions.firstRange();
+
+// If the upper bound is before the last sent position, we need to 
move ahead as these
+// individuallySentPositions are now irrelevant.
+if (range != null && 
range.upperEndpoint().compareTo(lastSentPosition) <= 0) {
+
individuallySentPositions.removeAtMost(lastSentPosition.getLedgerId(),
+lastSentPosition.getEntryId());
+range = individuallySentPositions.firstRange();
+}
+
+if (range != null) {
+// If the lowerBound is ahead of the last sent position,
+// verify if there are any entries in-between.
+if (range.lowerEndpoint().compareTo(lastSentPosition) <= 0 || 
managedLedger

Review Comment:
   > * Send `E6` to `c3`.
   > * State `replay-queue`: `E7, E8`, `sentPostionList`: `[E5~E6]`, 
`lastSentPistion`: `E6(re-calculate after sent)`
   
   The `sentPostionList` and `lastSentPistion` are not cleared by disconnecting 
a consumer. These are cleared when all consumers are disconnected.
   
https://github.com/equanz/pulsar/blob/c3b318af10b45d53c07cc2163a256ce98edf8e8b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java#L173-L179
   
   In this proposal (https://github.com/apache/pulsar/pull/20776) , we care 
only **not dispatched messages**. We don't care about already dispatched 
messages.
   
   * [issue-1]: Care about **read but not try to dispatched** messages
   * [issue-2]: Care about **skipped** messages
   
   I don't think we need to care about already dispatched messages. Because we 
can't control client-side redeliver operations from broker-side fully.
   
   simple example:
   (Assume that the selector will return `c2`)
   ```
   c1: 3:1, 3:2
   c2:
   lastSentPosition: 3:2
   recentlyJoinedConsumers: {c3: 3:2}
   -> (unack 3:2)
   c1: 3:1
   c2: 3:2
   lastSentPosition: 3:2
   recentlyJoinedConsumers: {c3: 3:2}
   -> (unack 3:1)
   c1:
   c2: 3:2, 3:1  // out-of-order
   lastSentPosition: 3:2
   recentlyJoinedConsumers: {c3: 3:2}
   ```
   
   https://github.com/apache/pulsar/pull/21953#discussion_r1519261144



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] [Bug] parseMessageMetadata error when broker entry metadata enable with high loading [pulsar]



lhotari commented on issue #22601:
URL: https://github.com/apache/pulsar/issues/22601#issuecomment-2100050623

   @semistone since you have some way to reproduce this in your own tests, 
would you be able to test if this can be reproduced with 
`dispatcherDispatchMessagesInSubscriptionThread=false`?
   
https://github.com/apache/pulsar/blob/80d46758e89b088688d521aa8ae401bfb00c98b2/conf/broker.conf#L435-L436
   
   It impacts this code:
   
https://github.com/apache/pulsar/blob/188355b2df08cafd9402e75baf1164ba4b44a052/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java#L659-L681


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(pulsar-site) branch main updated: Fix version in pulsar-admin-client dependency example

2024-05-08 Thread lhotari

This is an automated email from the ASF dual-hosted git repository.

lhotari pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/pulsar-site.git


The following commit(s) were added to refs/heads/main by this push:
 new fa8a58496c3a Fix version in pulsar-admin-client dependency example
fa8a58496c3a is described below

commit fa8a58496c3a477c0ce552b6d58232a1821e93b5
Author: Lari Hotari 
AuthorDate: Wed May 8 11:19:57 2024 +0300

Fix version in pulsar-admin-client dependency example
---
 docs/admin-get-started.md | 2 +-
 versioned_docs/version-3.0.x/admin-get-started.md | 4 ++--
 versioned_docs/version-3.2.x/admin-get-started.md | 4 ++--
 3 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/docs/admin-get-started.md b/docs/admin-get-started.md
index 3114c6292359..3e1ccc63cf71 100644
--- a/docs/admin-get-started.md
+++ b/docs/admin-get-started.md
@@ -427,7 +427,7 @@ To manage topics using Java admin API, complete following 
steps.
   
 org.apache.pulsar
 pulsar-client-admin
-2.11.0
+@pulsar:version@
 
   ```
 
diff --git a/versioned_docs/version-3.0.x/admin-get-started.md 
b/versioned_docs/version-3.0.x/admin-get-started.md
index dc8b3cde0cc0..bc21620f44bc 100644
--- a/versioned_docs/version-3.0.x/admin-get-started.md
+++ b/versioned_docs/version-3.0.x/admin-get-started.md
@@ -426,7 +426,7 @@ This tutorial guides you through every step of using Java 
admin API to manage to
   
 org.apache.pulsar
 pulsar-client-admin
-2.11.0
+@pulsar:version@
 
   ```
 
@@ -525,4 +525,4 @@ This tutorial guides you through every step of using Java 
admin API to manage to
 
 - [REST API](reference-rest-api-overview.md)
 
-- [Java admin API](/api/admin/)
\ No newline at end of file
+- [Java admin API](/api/admin/)
diff --git a/versioned_docs/version-3.2.x/admin-get-started.md 
b/versioned_docs/version-3.2.x/admin-get-started.md
index 3114c6292359..6767d39cb6ec 100644
--- a/versioned_docs/version-3.2.x/admin-get-started.md
+++ b/versioned_docs/version-3.2.x/admin-get-started.md
@@ -427,7 +427,7 @@ To manage topics using Java admin API, complete following 
steps.
   
 org.apache.pulsar
 pulsar-client-admin
-2.11.0
+@pulsar:version@
 
   ```
 
@@ -526,4 +526,4 @@ To manage topics using Java admin API, complete following 
steps.
 
 - [REST API](reference-rest-api-overview.md)
 
-- [Java admin API](/api/admin/)
\ No newline at end of file
+- [Java admin API](/api/admin/)

(pulsar) branch branch-3.3 created (now 188355b2df0)

2024-05-08 Thread zhaocong

This is an automated email from the ASF dual-hosted git repository.

zhaocong pushed a change to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/pulsar.git


  at 188355b2df0 [fix][admin] Fix deprecated check (#22653)

No new revisions were added by this update.

Re: [PR] [fix][sec] Upgrade postgresql version to avoid CVE-2024-1597 [pulsar]



codecov-commenter commented on PR #22635:
URL: https://github.com/apache/pulsar/pull/22635#issuecomment-2099972317

   ## 
[Codecov](https://app.codecov.io/gh/apache/pulsar/pull/22635?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
 Report
   All modified and coverable lines are covered by tests :white_check_mark:
   > Project coverage is 72.68%. Comparing base 
[(`bbc6224`)](https://app.codecov.io/gh/apache/pulsar/commit/bbc62245c5ddba1de4b1e7cee4ab49334bc36277?dropdown=coverage=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
 to head 
[(`82c1102`)](https://app.codecov.io/gh/apache/pulsar/pull/22635?dropdown=coverage=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache).
   > Report is 240 commits behind head on master.
   
   Additional details and impacted files
   
   
   [![Impacted file tree 
graph](https://app.codecov.io/gh/apache/pulsar/pull/22635/graphs/tree.svg?width=650=150=pr=acYqCpsK9J_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)](https://app.codecov.io/gh/apache/pulsar/pull/22635?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
   
   ```diff
   @@ Coverage Diff  @@
   ## master   #22635  +/-   ##
   
   - Coverage 73.57%   72.68%   -0.89% 
   + Complexity3262432329 -295 
   
 Files  1877 1887  +10 
 Lines139502   141018+1516 
 Branches  1529915480 +181 
   
   - Hits 102638   102502 -136 
   - Misses2890830658+1750 
   + Partials   7956 7858  -98 
   ```
   
   | 
[Flag](https://app.codecov.io/gh/apache/pulsar/pull/22635/flags?src=pr=flags_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
 | Coverage Δ | |
   |---|---|---|
   | 
[inttests](https://app.codecov.io/gh/apache/pulsar/pull/22635/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
 | `27.36% <ø> (+2.77%)` | :arrow_up: |
   | 
[systests](https://app.codecov.io/gh/apache/pulsar/pull/22635/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
 | `24.55% <ø> (+0.22%)` | :arrow_up: |
   | 
[unittests](https://app.codecov.io/gh/apache/pulsar/pull/22635/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
 | `71.46% <ø> (-1.39%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click 
here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#carryforward-flags-in-the-pull-request-comment)
 to find out more.
   
   [see 343 files with indirect coverage 
changes](https://app.codecov.io/gh/apache/pulsar/pull/22635/indirect-changes?src=pr=tree-more_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(pulsar) branch 3.3.0 deleted (was 188355b2df0)

2024-05-08 Thread zhaocong

This is an automated email from the ASF dual-hosted git repository.

zhaocong pushed a change to branch 3.3.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git


 was 188355b2df0 [fix][admin] Fix deprecated check (#22653)

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.

(pulsar) branch 3.3.0 created (now 188355b2df0)

2024-05-08 Thread zhaocong

This is an automated email from the ASF dual-hosted git repository.

zhaocong pushed a change to branch 3.3.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git


  at 188355b2df0 [fix][admin] Fix deprecated check (#22653)

No new revisions were added by this update.

[PR] [fix][broker] usedLocallySinceLastReport should always be reset [pulsar]

nodece opened a new pull request, #22672:
URL: https://github.com/apache/pulsar/pull/22672

### Motivation

`org.apache.pulsar.broker.resourcegroup.ResourceGroup.PerMonitoringClassFields#usedLocallySinceLastReport`
holds the usage rate of ResourceGroup in the current cycle.

When the report thread calls the
`org.apache.pulsar.broker.resourcegroup.ResourceGroup#rgFillResourceUsage` to
report the ResourceGroup usage, the `usedLocallySinceLastReport` should be
reset.

Right now, this value will be reported every 2 cycles. The correct behavior
should be to report the current rate only when it exceeds or falls below 5% of
the last reported value.

https://github.com/apache/pulsar/pull/22340 breaks this behavior.

### Modifications

- Always reset
`org.apache.pulsar.broker.resourcegroup.ResourceGroup.PerMonitoringClassFields#usedLocallySinceLastReport`
in the
`org.apache.pulsar.broker.resourcegroup.ResourceGroup#setUsageInMonitoredEntity`.

### Verifying this change

The test has been added.

### Documentation

- [ ] `doc`
- [ ] `doc-required`
- [x] `doc-not-needed`
- [ ] `doc-complete`

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] fix: fix print message id without batch index [pulsar-client-go]



Chenxulin97 commented on code in PR #1211:
URL: https://github.com/apache/pulsar-client-go/pull/1211#discussion_r1593476507


##
pulsar/impl_message.go:
##
@@ -184,7 +184,7 @@ func (id *messageID) BatchSize() int32 {
 }
 
 func (id *messageID) String() string {
-   return fmt.Sprintf("%d:%d:%d", id.ledgerID, id.entryID, id.partitionIdx)
+   return fmt.Sprintf("%d:%d:%d:%d", id.ledgerID, id.entryID, 
id.partitionIdx, id.batchIdx)

Review Comment:
   ok



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] [Bug] parseMessageMetadata error when broker entry metadata enable with high loading [pulsar]