(pulsar) branch branch-3.3 updated: [fix] [conf] fix configuration name and typo. (#22822)

2024-06-05 Thread technoboy 
This is an automated email from the ASF dual-hosted git repository.

technoboy pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/branch-3.3 by this push:
 new 13ee738fa2d [fix] [conf] fix configuration name and typo. (#22822)
13ee738fa2d is described below

commit 13ee738fa2d7268440c2d217f927b0810911f600
Author: Wenzhi Feng <52550727+thetumb...@users.noreply.github.com>
AuthorDate: Tue Jun 4 17:12:31 2024 +0800

[fix] [conf] fix configuration name and typo. (#22822)
---
 deployment/terraform-ansible/templates/broker.conf | 20 ++--
 1 file changed, 10 insertions(+), 10 deletions(-)

diff --git a/deployment/terraform-ansible/templates/broker.conf 
b/deployment/terraform-ansible/templates/broker.conf
index fe3bae6bb15..291cdd92147 100644
--- a/deployment/terraform-ansible/templates/broker.conf
+++ b/deployment/terraform-ansible/templates/broker.conf
@@ -320,7 +320,7 @@ dispatcherMinReadBatchSize=1
 # Max number of entries to dispatch for a shared subscription. By default it 
is 20 entries.
 dispatcherMaxRoundRobinBatchSize=20
 
-# Precise dispathcer flow control according to history message number of each 
entry
+# Precise dispatcher flow control according to history message number of each 
entry
 preciseDispatcherFlowControl=false
 
 # Max number of concurrent lookup request broker allows to throttle heavy 
incoming lookup traffic
@@ -638,7 +638,7 @@ bookkeeperMetadataServiceUri=
 # Authentication plugin to use when connecting to bookies
 bookkeeperClientAuthenticationPlugin=
 
-# BookKeeper auth plugin implementatation specifics parameters name and values
+# BookKeeper auth plugin implementation specifics parameters name and values
 bookkeeperClientAuthenticationParametersName=
 bookkeeperClientAuthenticationParameters=
 
@@ -944,7 +944,7 @@ defaultNamespaceBundleSplitAlgorithm=range_equally_divide
 
loadBalancerLoadSheddingStrategy=org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder
 
 # The broker resource usage threshold.
-# When the broker resource usage is gratter than the pulsar cluster average 
resource usge,
+# When the broker resource usage is greater than the pulsar cluster average 
resource usge,
 # the threshold shedder will be triggered to offload bundles from the broker.
 # It only take effect in ThresholdShedder strategy.
 loadBalancerBrokerThresholdShedderPercentage=10
@@ -953,27 +953,27 @@ loadBalancerBrokerThresholdShedderPercentage=10
 # It only take effect in ThresholdShedder strategy.
 loadBalancerHistoryResourcePercentage=0.9
 
-# The BandWithIn usage weight when calculating new resourde usage.
+# The BandWithIn usage weight when calculating new resource usage.
 # It only take effect in ThresholdShedder strategy.
 loadBalancerBandwithInResourceWeight=1.0
 
-# The BandWithOut usage weight when calculating new resourde usage.
+# The BandWithOut usage weight when calculating new resource usage.
 # It only take effect in ThresholdShedder strategy.
 loadBalancerBandwithOutResourceWeight=1.0
 
-# The CPU usage weight when calculating new resourde usage.
+# The CPU usage weight when calculating new resource usage.
 # It only take effect in ThresholdShedder strategy.
 loadBalancerCPUResourceWeight=1.0
 
-# The heap memory usage weight when calculating new resourde usage.
+# The heap memory usage weight when calculating new resource usage.
 # It only take effect in ThresholdShedder strategy.
 loadBalancerMemoryResourceWeight=1.0
 
-# The direct memory usage weight when calculating new resourde usage.
+# The direct memory usage weight when calculating new resource usage.
 # It only take effect in ThresholdShedder strategy.
 loadBalancerDirectMemoryResourceWeight=1.0
 
-# Bundle unload minimum throughput threshold (MB), avoding bundle unload 
frequently.
+# Bundle unload minimum throughput threshold (MB), avoiding bundle unload 
frequently.
 # It only take effect in ThresholdShedder strategy.
 loadBalancerBundleUnloadMinThroughputThreshold=10
 
@@ -995,7 +995,7 @@ replicatorPrefix=pulsar.repl
 
 # Duration to check replication policy to avoid replicator inconsistency
 # due to missing ZooKeeper watch (disable with value 0)
-replicatioPolicyCheckDurationSeconds=600
+replicationPolicyCheckDurationSeconds=600
 
 # Default message retention time. 0 means retention is disabled. -1 means data 
is not removed by time quota
 defaultRetentionTimeInMinutes=0

(pulsar) branch branch-3.3 updated: [improve] Validate range of argument before long -> int conversion (#22830)

2024-06-05 Thread technoboy 
This is an automated email from the ASF dual-hosted git repository.

technoboy pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/branch-3.3 by this push:
 new cabed9a92da [improve] Validate range of argument before long -> int 
conversion (#22830)
cabed9a92da is described below

commit cabed9a92da6992fa706aeee4422be385ab9a2a6
Author: Matteo Merli 
AuthorDate: Tue Jun 4 03:12:33 2024 -0700

[improve] Validate range of argument before long -> int conversion (#22830)
---
 .../org/apache/pulsar/broker/admin/impl/TransactionsBase.java| 9 -
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/TransactionsBase.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/TransactionsBase.java
index 1014c9fe8e3..4fef0802ed4 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/TransactionsBase.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/TransactionsBase.java
@@ -564,8 +564,15 @@ public abstract class TransactionsBase extends 
AdminResource {
 
 protected CompletableFuture internalAbortTransaction(boolean 
authoritative, long mostSigBits,
long 
leastSigBits) {
+
+if (mostSigBits < 0 || mostSigBits > Integer.MAX_VALUE) {
+return CompletableFuture.failedFuture(new 
IllegalArgumentException("mostSigBits out of bounds"));
+}
+
+int partitionIdx = (int) mostSigBits;
+
 return validateTopicOwnershipAsync(
-
SystemTopicNames.TRANSACTION_COORDINATOR_ASSIGN.getPartition((int) 
mostSigBits), authoritative)
+
SystemTopicNames.TRANSACTION_COORDINATOR_ASSIGN.getPartition(partitionIdx), 
authoritative)
 .thenCompose(__ -> validateSuperUserAccessAsync())
 .thenCompose(__ -> 
pulsar().getTransactionMetadataStoreService()
 .endTransaction(new TxnID(mostSigBits, leastSigBits), 
TxnAction.ABORT_VALUE, false));

(pulsar) branch branch-3.3 updated: [fix][meta] Check if metadata store is closed in RocksdbMetadataStore (#22852)

2024-06-05 Thread technoboy 
This is an automated email from the ASF dual-hosted git repository.

technoboy pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/branch-3.3 by this push:
 new 37f6239b5c0 [fix][meta] Check if metadata store is closed in 
RocksdbMetadataStore (#22852)
37f6239b5c0 is described below

commit 37f6239b5c0e9f03169ad02bae0e0299e6107342
Author: Lari Hotari 
AuthorDate: Wed Jun 5 22:02:43 2024 +0300

[fix][meta] Check if metadata store is closed in RocksdbMetadataStore 
(#22852)
---
 .../metadata/impl/AbstractMetadataStore.java   | 25 +++---
 .../pulsar/metadata/impl/RocksdbMetadataStore.java | 15 +
 2 files changed, 27 insertions(+), 13 deletions(-)

diff --git 
a/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/AbstractMetadataStore.java
 
b/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/AbstractMetadataStore.java
index 0a356643914..77fd21f1342 100644
--- 
a/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/AbstractMetadataStore.java
+++ 
b/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/AbstractMetadataStore.java
@@ -257,8 +257,7 @@ public abstract class AbstractMetadataStore implements 
MetadataStoreExtended, Co
 @Override
 public CompletableFuture> get(String path) {
 if (isClosed()) {
-return FutureUtil.failedFuture(
-new MetadataStoreException.AlreadyClosedException());
+return alreadyClosedFailedFuture();
 }
 long start = System.currentTimeMillis();
 if (!isValidPath(path)) {
@@ -286,8 +285,7 @@ public abstract class AbstractMetadataStore implements 
MetadataStoreExtended, Co
 @Override
 public final CompletableFuture> getChildren(String path) {
 if (isClosed()) {
-return FutureUtil.failedFuture(
-new MetadataStoreException.AlreadyClosedException());
+return alreadyClosedFailedFuture();
 }
 if (!isValidPath(path)) {
 return FutureUtil.failedFuture(new 
MetadataStoreException.InvalidPathException(path));
@@ -298,8 +296,7 @@ public abstract class AbstractMetadataStore implements 
MetadataStoreExtended, Co
 @Override
 public final CompletableFuture exists(String path) {
 if (isClosed()) {
-return FutureUtil.failedFuture(
-new MetadataStoreException.AlreadyClosedException());
+return alreadyClosedFailedFuture();
 }
 if (!isValidPath(path)) {
 return FutureUtil.failedFuture(new 
MetadataStoreException.InvalidPathException(path));
@@ -361,8 +358,7 @@ public abstract class AbstractMetadataStore implements 
MetadataStoreExtended, Co
 @Override
 public final CompletableFuture delete(String path, Optional 
expectedVersion) {
 if (isClosed()) {
-return FutureUtil.failedFuture(
-new MetadataStoreException.AlreadyClosedException());
+return alreadyClosedFailedFuture();
 }
 long start = System.currentTimeMillis();
 if (!isValidPath(path)) {
@@ -411,8 +407,7 @@ public abstract class AbstractMetadataStore implements 
MetadataStoreExtended, Co
 @Override
 public CompletableFuture deleteRecursive(String path) {
 if (isClosed()) {
-return FutureUtil.failedFuture(
-new MetadataStoreException.AlreadyClosedException());
+return alreadyClosedFailedFuture();
 }
 return getChildren(path)
 .thenCompose(children -> FutureUtil.waitForAll(
@@ -436,8 +431,7 @@ public abstract class AbstractMetadataStore implements 
MetadataStoreExtended, Co
 public final CompletableFuture put(String path, byte[] data, 
Optional optExpectedVersion,
 EnumSet options) {
 if (isClosed()) {
-return FutureUtil.failedFuture(
-new MetadataStoreException.AlreadyClosedException());
+return alreadyClosedFailedFuture();
 }
 long start = System.currentTimeMillis();
 if (!isValidPath(path)) {
@@ -517,10 +511,15 @@ public abstract class AbstractMetadataStore implements 
MetadataStoreExtended, Co
 }
 }
 
-private boolean isClosed() {
+protected boolean isClosed() {
 return isClosed.get();
 }
 
+protected static  CompletableFuture alreadyClosedFailedFuture() {
+return FutureUtil.failedFuture(
+new MetadataStoreException.AlreadyClosedException());
+}
+
 @Override
 public void close() throws Exception {
 executor.shutdownNow();
diff --git 
a/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/RocksdbMetadataStore.java
 
b/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/RocksdbMetadataStore.java
index 39f7edd5cee..06f7b260536 100644
---

(pulsar) branch branch-3.3 updated: [improve] [broker] Do not call cursor.isCursorDataFullyPersistable if disabled dispatcherPauseOnAckStatePersistentEnabled (#22729)

2024-06-05 Thread technoboy 
This is an automated email from the ASF dual-hosted git repository.

technoboy pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/branch-3.3 by this push:
 new b33fc099e7d [improve] [broker] Do not call 
cursor.isCursorDataFullyPersistable if disabled 
dispatcherPauseOnAckStatePersistentEnabled (#22729)
b33fc099e7d is described below

commit b33fc099e7d663e1b93c6ca8ec95007187dbf8e7
Author: fengyubiao 
AuthorDate: Fri May 17 13:50:48 2024 +0800

[improve] [broker] Do not call cursor.isCursorDataFullyPersistable if 
disabled dispatcherPauseOnAckStatePersistentEnabled (#22729)
---
 .../PersistentDispatcherMultipleConsumers.java |  9 
 .../api/SubscriptionPauseOnAckStatPersistTest.java | 50 ++
 2 files changed, 59 insertions(+)

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java
index 49a19c0fe31..f20750fa0c2 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java
@@ -1087,6 +1087,15 @@ public class PersistentDispatcherMultipleConsumers 
extends AbstractDispatcherMul
 @Override
 public boolean checkAndResumeIfPaused() {
 boolean paused = blockedDispatcherOnCursorDataCanNotFullyPersist == 
TRUE;
+// Calling "cursor.isCursorDataFullyPersistable()" will loop the 
collection "individualDeletedMessages". It is
+// not a light method.
+// If never enabled "dispatcherPauseOnAckStatePersistentEnabled", skip 
the following checks to improve
+// performance.
+if (!paused && !topic.isDispatcherPauseOnAckStatePersistentEnabled()){
+// "true" means no need to pause.
+return true;
+}
+// Enabled "dispatcherPauseOnAckStatePersistentEnabled" before.
 boolean shouldPauseNow = !cursor.isCursorDataFullyPersistable()
 && topic.isDispatcherPauseOnAckStatePersistentEnabled();
 // No need to change.
diff --git 
a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SubscriptionPauseOnAckStatPersistTest.java
 
b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SubscriptionPauseOnAckStatPersistTest.java
index 9a4de8ecf21..36c36735c06 100644
--- 
a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SubscriptionPauseOnAckStatPersistTest.java
+++ 
b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SubscriptionPauseOnAckStatPersistTest.java
@@ -23,8 +23,12 @@ import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Map;
 import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
 import lombok.extern.slf4j.Slf4j;
 import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedCursorContainer;
+import org.apache.bookkeeper.mledger.impl.ManagedCursorImpl;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
 import org.apache.pulsar.broker.BrokerTestUtil;
 import org.apache.pulsar.broker.service.Dispatcher;
 import org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService;
@@ -38,6 +42,7 @@ import 
org.apache.pulsar.common.policies.data.HierarchyTopicPolicies;
 import org.apache.pulsar.common.policies.data.TopicPolicies;
 import org.awaitility.Awaitility;
 import org.awaitility.reflect.WhiteboxImpl;
+import org.mockito.Mockito;
 import org.testng.Assert;
 import org.testng.annotations.AfterClass;
 import org.testng.annotations.BeforeClass;
@@ -549,4 +554,49 @@ public class SubscriptionPauseOnAckStatPersistTest extends 
ProducerConsumerBase
 c4.close();
 admin.topics().delete(tpName, false);
 }
+
+@Test(dataProvider = "multiConsumerSubscriptionTypes")
+public void 
testNeverCallCursorIsCursorDataFullyPersistableIfDisabledTheFeature(SubscriptionType
 subscriptionType)
+throws Exception {
+final String tpName = 
BrokerTestUtil.newUniqueName("persistent://public/default/tp");
+final String mlName = 
TopicName.get(tpName).getPersistenceNamingEncoding();
+final String subscription = "s1";
+final int msgSendCount = 100;
+// Inject a injection to record the counter of calling 
"cursor.isCursorDataFullyPersistable".
+final ManagedLedgerImpl ml = (ManagedLedgerImpl) 
pulsar.getBrokerService().getManagedLedgerFactory().open(mlName);
+final ManagedCursorImpl cursor = (ManagedCursorImpl) 
ml.openCursor(subscription);
+final ManagedCursorImpl spyCursor = Mockito.spy(cursor);
+AtomicInteger callingIsCursorDataFullyPersistableCounter = new 
AtomicInteger();
+

(pulsar) branch branch-3.3 updated: [fix][ml]: subscription props could be lost in case of missing ledger during recovery (#22637)

2024-06-05 Thread technoboy 
This is an automated email from the ASF dual-hosted git repository.

technoboy pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/branch-3.3 by this push:
 new 71d62b2613b [fix][ml]: subscription props could be lost in case of 
missing ledger during recovery (#22637)
71d62b2613b is described below

commit 71d62b2613bbb7f378848e308e1616b65d223169
Author: Nicolò Boschi 
AuthorDate: Mon May 20 18:12:21 2024 +0200

[fix][ml]: subscription props could be lost in case of missing ledger 
during recovery (#22637)
---
 .../main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
 
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
index 35000361eca..18d9cd7cb05 100644
--- 
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
+++ 
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
@@ -543,7 +543,7 @@ public class ManagedCursorImpl implements ManagedCursor {
 log.error("[{}] Error opening metadata ledger {} for cursor 
{}: {}", ledger.getName(), ledgerId, name,
 BKException.getMessage(rc));
 // Rewind to oldest entry available
-initialize(getRollbackPosition(info), Collections.emptyMap(), 
Collections.emptyMap(), callback);
+initialize(getRollbackPosition(info), Collections.emptyMap(), 
cursorProperties, callback);
 return;
 } else if (rc != BKException.Code.OK) {
 log.warn("[{}] Error opening metadata ledger {} for cursor {}: 
{}", ledger.getName(), ledgerId, name,

(pulsar) branch branch-3.3 updated: [improve][broker] Close protocol handlers before unloading namespace bundles (#22728)

2024-06-05 Thread technoboy 
This is an automated email from the ASF dual-hosted git repository.

technoboy pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/branch-3.3 by this push:
 new d608073edab [improve][broker] Close protocol handlers before unloading 
namespace bundles (#22728)
d608073edab is described below

commit d608073edab5938f0274193b676f6fced21a82f2
Author: Yunze Xu 
AuthorDate: Tue May 21 16:26:36 2024 +0800

[improve][broker] Close protocol handlers before unloading namespace 
bundles (#22728)
---
 .../org/apache/pulsar/broker/PulsarService.java|  12 +-
 .../channel/ServiceUnitStateChannelImpl.java   |   2 +-
 .../broker/protocol/PulsarClientBasedHandler.java  | 152 +
 .../protocol/PulsarClientBasedHandlerTest.java |  87 
 .../protocol/SimpleProtocolHandlerTestsBase.java   |  16 ++-
 5 files changed, 258 insertions(+), 11 deletions(-)

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
index ac37aca531a..db1e1dbe402 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
@@ -442,6 +442,12 @@ public class PulsarService implements AutoCloseable, 
ShutdownService {
 public CompletableFuture closeAsync() {
 mutex.lock();
 try {
+// Close protocol handler before unloading namespace bundles 
because protocol handlers might maintain
+// Pulsar clients that could send lookup requests that affect 
unloading.
+if (protocolHandlers != null) {
+protocolHandlers.close();
+protocolHandlers = null;
+}
 if (closeFuture != null) {
 return closeFuture;
 }
@@ -449,6 +455,7 @@ public class PulsarService implements AutoCloseable, 
ShutdownService {
 if (brokerService != null) {
 brokerService.unloadNamespaceBundlesGracefully();
 }
+// It only tells the Pulsar clients that this service is not ready 
to serve for the lookup requests
 state = State.Closing;
 
 // close the service in reverse order v.s. in which they are 
started
@@ -510,11 +517,6 @@ public class PulsarService implements AutoCloseable, 
ShutdownService {
 (long) 
(GRACEFUL_SHUTDOWN_TIMEOUT_RATIO_OF_TOTAL_TIMEOUT
 * getConfiguration()
 
.getBrokerShutdownTimeoutMs(;
-// close protocol handler before closing broker service
-if (protocolHandlers != null) {
-protocolHandlers.close();
-protocolHandlers = null;
-}
 
 // cancel loadShedding task and shutdown the loadManager executor 
before shutting down the broker
 cancelLoadBalancerTasks();
diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java
index bf6266482f8..5e3d1c78361 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java
@@ -114,7 +114,7 @@ public class ServiceUnitStateChannelImpl implements 
ServiceUnitStateChannel {
 public static final CompressionType MSG_COMPRESSION_TYPE = 
CompressionType.ZSTD;
 private static final int OWNERSHIP_CLEAN_UP_MAX_WAIT_TIME_IN_MILLIS = 5000;
 private static final int OWNERSHIP_CLEAN_UP_WAIT_RETRY_DELAY_IN_MILLIS = 
100;
-private static final int OWNERSHIP_CLEAN_UP_CONVERGENCE_DELAY_IN_MILLIS = 
3000;
+public static final int OWNERSHIP_CLEAN_UP_CONVERGENCE_DELAY_IN_MILLIS = 
3000;
 public static final long VERSION_ID_INIT = 1; // initial versionId
 public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 
mins
 private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs 
to clean immediately
diff --git 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/protocol/PulsarClientBasedHandler.java
 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/protocol/PulsarClientBasedHandler.java
new file mode 100644
index 000..ed9881a8cad
--- /dev/null
+++ 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/protocol/PulsarClientBasedHandler.java
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work

(pulsar) branch branch-3.3 updated: [improve][offload] Allow to disable the managedLedgerOffloadDeletionLagInMillis (#22689)

2024-06-05 Thread technoboy 
This is an automated email from the ASF dual-hosted git repository.

technoboy pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/branch-3.3 by this push:
 new 4e467091ddf [improve][offload] Allow to disable the 
managedLedgerOffloadDeletionLagInMillis (#22689)
4e467091ddf is described below

commit 4e467091ddfbb9f164cf45715dceb3b3d72ea974
Author: Yong Zhang 
AuthorDate: Thu May 16 20:54:32 2024 +0800

[improve][offload] Allow to disable the 
managedLedgerOffloadDeletionLagInMillis (#22689)
---
 .../java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java| 3 ++-
 .../org/apache/bookkeeper/mledger/impl/OffloadLedgerDeleteTest.java   | 4 
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git 
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
 
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
index 167c0a1bad3..1e3e7e065d7 100644
--- 
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
+++ 
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
@@ -2592,6 +2592,7 @@ public class ManagedLedgerImpl implements ManagedLedger, 
CreateCallback {
 long elapsedMs = clock.millis() - offload.getTimestamp();
 return offloadPolicies.filter(policies -> offload.getComplete() && 
!offload.getBookkeeperDeleted()
 && policies.getManagedLedgerOffloadDeletionLagInMillis() != 
null
+&& policies.getManagedLedgerOffloadDeletionLagInMillis() >= 0
 && elapsedMs > 
policies.getManagedLedgerOffloadDeletionLagInMillis()).isPresent();
 }
 
@@ -4574,4 +4575,4 @@ public class ManagedLedgerImpl implements ManagedLedger, 
CreateCallback {
 }
 return theSlowestNonDurableReadPosition;
 }
-}
\ No newline at end of file
+}
diff --git 
a/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/OffloadLedgerDeleteTest.java
 
b/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/OffloadLedgerDeleteTest.java
index 56da315553e..b46f06106cf 100644
--- 
a/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/OffloadLedgerDeleteTest.java
+++ 
b/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/OffloadLedgerDeleteTest.java
@@ -383,6 +383,10 @@ public class OffloadLedgerDeleteTest extends 
MockedBookKeeperTestCase {
 needsDelete = managedLedger.isOffloadedNeedsDelete(offloadContext, 
Optional.of(offloadPolicies));
 Assert.assertTrue(needsDelete);
 
+offloadPolicies.setManagedLedgerOffloadDeletionLagInMillis(-1L);
+needsDelete = managedLedger.isOffloadedNeedsDelete(offloadContext, 
Optional.of(offloadPolicies));
+Assert.assertFalse(needsDelete);
+
 offloadPolicies.setManagedLedgerOffloadDeletionLagInMillis(1000L * 2);
 needsDelete = managedLedger.isOffloadedNeedsDelete(offloadContext, 
Optional.of(offloadPolicies));
 Assert.assertFalse(needsDelete);

(pulsar) branch branch-3.3 updated: [fix][broker] usedLocallySinceLastReport should always be reset (#22672)

2024-06-05 Thread technoboy 
This is an automated email from the ASF dual-hosted git repository.

technoboy pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/branch-3.3 by this push:
 new 00675bcea1f [fix][broker] usedLocallySinceLastReport should always be 
reset (#22672)
00675bcea1f is described below

commit 00675bcea1facaf1fe6942476186f3449f5c8419
Author: Zixuan Liu 
AuthorDate: Thu May 9 09:42:17 2024 +0800

[fix][broker] usedLocallySinceLastReport should always be reset (#22672)

Signed-off-by: Zixuan Liu 
---
 .../pulsar/broker/resourcegroup/ResourceGroup.java |  3 +-
 .../ResourceGroupReportLocalUsageTest.java | 50 ++
 2 files changed, 34 insertions(+), 19 deletions(-)

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/resourcegroup/ResourceGroup.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/resourcegroup/ResourceGroup.java
index f8ec52bfe3c..541a645f18b 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/resourcegroup/ResourceGroup.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/resourcegroup/ResourceGroup.java
@@ -458,14 +458,13 @@ public class ResourceGroup {
 
 bytesUsed = monEntity.usedLocallySinceLastReport.bytes;
 messagesUsed = monEntity.usedLocallySinceLastReport.messages;
-
+monEntity.usedLocallySinceLastReport.bytes = 
monEntity.usedLocallySinceLastReport.messages = 0;
 if (sendReport) {
 p.setBytesPerPeriod(bytesUsed);
 p.setMessagesPerPeriod(messagesUsed);
 monEntity.lastReportedValues.bytes = bytesUsed;
 monEntity.lastReportedValues.messages = messagesUsed;
 monEntity.numSuppressedUsageReports = 0;
-monEntity.usedLocallySinceLastReport.bytes = 
monEntity.usedLocallySinceLastReport.messages = 0;
 monEntity.totalUsedLocally.bytes += bytesUsed;
 monEntity.totalUsedLocally.messages += messagesUsed;
 monEntity.lastResourceUsageFillTimeMSecsSinceEpoch = 
System.currentTimeMillis();
diff --git 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/resourcegroup/ResourceGroupReportLocalUsageTest.java
 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/resourcegroup/ResourceGroupReportLocalUsageTest.java
index 658b7c94165..139d19886c7 100644
--- 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/resourcegroup/ResourceGroupReportLocalUsageTest.java
+++ 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/resourcegroup/ResourceGroupReportLocalUsageTest.java
@@ -72,34 +72,50 @@ public class ResourceGroupReportLocalUsageTest extends 
MockedPulsarServiceBaseTe
 rgConfig.setPublishRateInMsgs(2000);
 service.resourceGroupCreate(rgName, rgConfig);
 
-org.apache.pulsar.broker.resourcegroup.ResourceGroup resourceGroup = 
service.resourceGroupGet(rgName);
 BytesAndMessagesCount bytesAndMessagesCount = new 
BytesAndMessagesCount();
 bytesAndMessagesCount.bytes = 20;
 bytesAndMessagesCount.messages = 10;
-
resourceGroup.incrementLocalUsageStats(ResourceGroupMonitoringClass.Publish, 
bytesAndMessagesCount);
+
+org.apache.pulsar.broker.resourcegroup.ResourceGroup resourceGroup = 
service.resourceGroupGet(rgName);
+for (ResourceGroupMonitoringClass value : 
ResourceGroupMonitoringClass.values()) {
+resourceGroup.incrementLocalUsageStats(value, 
bytesAndMessagesCount);
+}
+
+// Case1: Suppress report ResourceUsage.
+needReport.set(false);
 ResourceUsage resourceUsage = new ResourceUsage();
 resourceGroup.rgFillResourceUsage(resourceUsage);
 assertFalse(resourceUsage.hasDispatch());
 assertFalse(resourceUsage.hasPublish());
+for (ResourceGroupMonitoringClass value : 
ResourceGroupMonitoringClass.values()) {
+PerMonitoringClassFields monitoredEntity =
+resourceGroup.getMonitoredEntity(value);
+assertEquals(monitoredEntity.usedLocallySinceLastReport.messages, 
0);
+assertEquals(monitoredEntity.usedLocallySinceLastReport.bytes, 0);
+assertEquals(monitoredEntity.totalUsedLocally.messages, 0);
+assertEquals(monitoredEntity.totalUsedLocally.bytes, 0);
+assertEquals(monitoredEntity.lastReportedValues.messages, 0);
+assertEquals(monitoredEntity.lastReportedValues.bytes, 0);
+}
 
-PerMonitoringClassFields publishMonitoredEntity =
-
resourceGroup.getMonitoredEntity(ResourceGroupMonitoringClass.Publish);
-
assertEquals(publishMonitoredEntity.usedLocallySinceLastReport.messages, 
bytesAndMessagesCount.messages);
-assertEquals(publishMonitoredEntity.usedLocallySinceLastReport.bytes, 
bytesAndMessagesCount.bytes);
-

(pulsar) branch branch-3.3 updated: [fix] [broker] rename to changeMaxReadPositionCount (#22656)

2024-06-05 Thread technoboy 
This is an automated email from the ASF dual-hosted git repository.

technoboy pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/branch-3.3 by this push:
 new 4a1779e2b36 [fix] [broker] rename to changeMaxReadPositionCount 
(#22656)
4a1779e2b36 is described below

commit 4a1779e2b369867e47979362a576fd81b25c41e3
Author: Wenzhi Feng <52550727+thetumb...@users.noreply.github.com>
AuthorDate: Wed May 8 19:34:00 2024 +0800

[fix] [broker] rename to changeMaxReadPositionCount (#22656)
---
 .../transaction/buffer/impl/TopicTransactionBuffer.java  | 16 
 .../pulsar/broker/transaction/TransactionTest.java   | 12 ++--
 2 files changed, 14 insertions(+), 14 deletions(-)

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/TopicTransactionBuffer.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/TopicTransactionBuffer.java
index a36216bd625..81c9ecfc728 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/TopicTransactionBuffer.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/TopicTransactionBuffer.java
@@ -76,8 +76,8 @@ public class TopicTransactionBuffer extends 
TopicTransactionBufferState implemen
  */
 private final LinkedMap ongoingTxns = new 
LinkedMap<>();
 
-// when add abort or change max read position, the count will +1. Take 
snapshot will set 0 into it.
-private final AtomicLong changeMaxReadPositionAndAddAbortTimes = new 
AtomicLong();
+// when change max read position, the count will +1. Take snapshot will 
reset the count.
+private final AtomicLong changeMaxReadPositionCount = new AtomicLong();
 
 private final LongAdder txnCommittedCounter = new LongAdder();
 
@@ -429,15 +429,15 @@ public class TopicTransactionBuffer extends 
TopicTransactionBufferState implemen
 }
 
 private void takeSnapshotByChangeTimes() {
-if (changeMaxReadPositionAndAddAbortTimes.get() >= 
takeSnapshotIntervalNumber) {
-this.changeMaxReadPositionAndAddAbortTimes.set(0);
+if (changeMaxReadPositionCount.get() >= takeSnapshotIntervalNumber) {
+this.changeMaxReadPositionCount.set(0);
 
this.snapshotAbortedTxnProcessor.takeAbortedTxnsSnapshot(this.maxReadPosition);
 }
 }
 
 private void takeSnapshotByTimeout() {
-if (changeMaxReadPositionAndAddAbortTimes.get() > 0) {
-this.changeMaxReadPositionAndAddAbortTimes.set(0);
+if (changeMaxReadPositionCount.get() > 0) {
+this.changeMaxReadPositionCount.set(0);
 
this.snapshotAbortedTxnProcessor.takeAbortedTxnsSnapshot(this.maxReadPosition);
 }
 this.timer.newTimeout(TopicTransactionBuffer.this,
@@ -454,7 +454,7 @@ public class TopicTransactionBuffer extends 
TopicTransactionBufferState implemen
 maxReadPosition = (PositionImpl) 
topic.getManagedLedger().getLastConfirmedEntry();
 }
 if (preMaxReadPosition.compareTo(this.maxReadPosition) != 0) {
-this.changeMaxReadPositionAndAddAbortTimes.getAndIncrement();
+this.changeMaxReadPositionCount.getAndIncrement();
 }
 }
 
@@ -489,7 +489,7 @@ public class TopicTransactionBuffer extends 
TopicTransactionBufferState implemen
 } else if (checkIfReady()) {
 if (ongoingTxns.isEmpty()) {
 maxReadPosition = position;
-changeMaxReadPositionAndAddAbortTimes.incrementAndGet();
+changeMaxReadPositionCount.incrementAndGet();
 }
 }
 }
diff --git 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TransactionTest.java
 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TransactionTest.java
index e8c15d193a2..5e806bb9cee 100644
--- 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TransactionTest.java
+++ 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TransactionTest.java
@@ -1095,10 +1095,10 @@ public class TransactionTest extends 
TransactionTestBase {
 }
 
 @Test
-public void 
testNotChangeMaxReadPositionAndAddAbortTimesWhenCheckIfNoSnapshot() throws 
Exception {
+public void testNotChangeMaxReadPositionCountWhenCheckIfNoSnapshot() 
throws Exception {
 PersistentTopic persistentTopic = (PersistentTopic) 
getPulsarServiceList().get(0)
 .getBrokerService()
-.getTopic(NAMESPACE1 + 
"/changeMaxReadPositionAndAddAbortTimes" + UUID.randomUUID(), true)
+.getTopic(NAMESPACE1 + "/changeMaxReadPositionCount" + 
UUID.randomUUID(), true)
 .get().get();
 TransactionBuffer buffer = persistentTopic.getTransactionBuffer();
 Field processorField =

(pulsar) branch branch-3.3 updated: [fix][sec] Upgrade postgresql version to avoid CVE-2024-1597 (#22635)

2024-06-05 Thread technoboy 
This is an automated email from the ASF dual-hosted git repository.

technoboy pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/branch-3.3 by this push:
 new 72641cb1888 [fix][sec] Upgrade postgresql version to avoid 
CVE-2024-1597 (#22635)
72641cb1888 is described below

commit 72641cb18883bc03041029e656b805bb94463a79
Author: Nikhil Erigila <60037808+nikhilerigil...@users.noreply.github.com>
AuthorDate: Fri May 10 16:41:20 2024 +0530

[fix][sec] Upgrade postgresql version to avoid CVE-2024-1597 (#22635)
---
 pom.xml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/pom.xml b/pom.xml
index af45b297b8c..23e3d3a15e2 100644
--- a/pom.xml
+++ b/pom.xml
@@ -190,7 +190,7 @@ flexible messaging model and an intuitive client 
API.
 5.1.0
 3.42.0.0
 8.0.11
-42.5.1
+42.5.5
 0.4.6
 2.7.5
 0.4.4-hotfix1
@@ -199,7 +199,7 @@ flexible messaging model and an intuitive client 
API.
 1.2.4
 8.12.1
 1.9.7.Final
-42.5.0
+42.5.5
 8.0.30
 
 1.15.16.Final

(pulsar) branch branch-3.3 updated: [improve] [broker] Add additionalSystemCursorNames ignore list for TTL check (#22614)

2024-06-05 Thread technoboy 
This is an automated email from the ASF dual-hosted git repository.

technoboy pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/branch-3.3 by this push:
 new 79de2daf507 [improve] [broker] Add additionalSystemCursorNames ignore 
list for TTL check (#22614)
79de2daf507 is described below

commit 79de2daf50700dfb0ecd4fe00ee4af6af775a288
Author: Hang Chen 
AuthorDate: Thu May 9 20:45:56 2024 +0800

[improve] [broker] Add additionalSystemCursorNames ignore list for TTL 
check (#22614)
---
 conf/broker.conf   |  4 +
 conf/standalone.conf   |  4 +
 .../apache/pulsar/broker/ServiceConfiguration.java |  7 ++
 .../broker/service/persistent/PersistentTopic.java |  7 +-
 .../pulsar/broker/service/MessageTTLTest.java  | 96 ++
 5 files changed, 117 insertions(+), 1 deletion(-)

diff --git a/conf/broker.conf b/conf/broker.conf
index 1b51ff47551..1ef68a0395c 100644
--- a/conf/broker.conf
+++ b/conf/broker.conf
@@ -180,6 +180,10 @@ backlogQuotaDefaultRetentionPolicy=producer_request_hold
 # Default ttl for namespaces if ttl is not already configured at namespace 
policies. (disable default-ttl with value 0)
 ttlDurationDefaultInSeconds=0
 
+# Additional system subscriptions that will be ignored by ttl check. The 
cursor names are comma separated.
+# Default is empty.
+# additionalSystemCursorNames=
+
 # Enable topic auto creation if new producer or consumer connected (disable 
auto creation with value false)
 allowAutoTopicCreation=true
 
diff --git a/conf/standalone.conf b/conf/standalone.conf
index 51035235d4d..a8615b70293 100644
--- a/conf/standalone.conf
+++ b/conf/standalone.conf
@@ -121,6 +121,10 @@ backlogQuotaDefaultLimitSecond=-1
 # Default ttl for namespaces if ttl is not already configured at namespace 
policies. (disable default-ttl with value 0)
 ttlDurationDefaultInSeconds=0
 
+# Additional system subscriptions that will be ignored by ttl check.  The 
cursor names are comma separated.
+# Default is empty.
+# additionalSystemCursorNames=
+
 # Enable the deletion of inactive topics. This parameter need to cooperate 
with the allowAutoTopicCreation parameter.
 # If brokerDeleteInactiveTopicsEnabled is set to true, we should ensure that 
allowAutoTopicCreation is also set to true.
 brokerDeleteInactiveTopicsEnabled=true
diff --git 
a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
 
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
index a9d170ea5de..9efe1856509 100644
--- 
a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
+++ 
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
@@ -652,6 +652,13 @@ public class ServiceConfiguration implements 
PulsarConfiguration {
 )
 private int ttlDurationDefaultInSeconds = 0;
 
+@FieldContext(
+category = CATEGORY_POLICIES,
+doc = "Additional system subscriptions that will be ignored by ttl 
check. "
++ "The cursor names are comma separated. Default is empty."
+)
+private Set additionalSystemCursorNames = new TreeSet<>();
+
 @FieldContext(
 category = CATEGORY_POLICIES,
 dynamic = true,
diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
index 7228bdeb2d3..28bc27f7961 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
@@ -40,6 +40,7 @@ import java.util.Map;
 import java.util.Objects;
 import java.util.Optional;
 import java.util.Set;
+import java.util.TreeSet;
 import java.util.concurrent.CancellationException;
 import java.util.concurrent.CompletableFuture;
 import java.util.concurrent.CompletionException;
@@ -279,6 +280,7 @@ public class PersistentTopic extends AbstractTopic 
implements Topic, AddEntryCal
 private final ExecutorService orderedExecutor;
 
 private volatile CloseFutures closeFutures;
+private Set additionalSystemCursorNames = new TreeSet<>();
 
 @Getter
 private final PersistentTopicMetrics persistentTopicMetrics = new 
PersistentTopicMetrics();
@@ -414,6 +416,7 @@ public class PersistentTopic extends AbstractTopic 
implements Topic, AddEntryCal
 } else {
 shadowSourceTopic = null;
 }
+additionalSystemCursorNames = 
brokerService.pulsar().getConfiguration().getAdditionalSystemCursorNames();
 }
 
 @Override
@@ -1934,7 +1937,9 @@ public class PersistentTopic extends AbstractTopic 
implements Topic, AddEntryCal
 int messageTtlInSeconds =

(pulsar-site) branch release-3.3.0 updated (61be26ea50bc -> a6a718823d87)

2024-06-05 Thread zhaocong 
This is an automated email from the ASF dual-hosted git repository.

zhaocong pushed a change to branch release-3.3.0
in repository https://gitbox.apache.org/repos/asf/pulsar-site.git


from 61be26ea50bc Release 3.3.0
 add 4eee778c58e6 Docs sync done from apache/pulsar (#d74010c)
 add a6a718823d87 Merge branch 'main' into release-3.3.0

No new revisions were added by this update.

Summary of changes:
 static/reference/next/config/reference-configuration-broker.md | 2 +-
 static/reference/next/config/reference-configuration-standalone.md | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

[PR] Release 3.3.0 [pulsar-site]

2024-06-05 Thread via GitHub 


coderzc opened a new pull request, #910:
URL: https://github.com/apache/pulsar-site/pull/910

   ### ✅ Contribution Checklist
   
   
   
   - [ ] I read the [contribution 
guide](https://pulsar.apache.org/contribute/document-contribution/)
   - [x] I updated the [versioned 
docs](https://pulsar.apache.org/contribute/document-contribution/#update-versioned-docs)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(pulsar-site) branch release-3.3.0 created (now 61be26ea50bc)

2024-06-05 Thread zhaocong 
This is an automated email from the ASF dual-hosted git repository.

zhaocong pushed a change to branch release-3.3.0
in repository https://gitbox.apache.org/repos/asf/pulsar-site.git


  at 61be26ea50bc Release 3.3.0

This branch includes the following new commits:

 new 61be26ea50bc Release 3.3.0

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

[PR] [fix][doc] Fix the doc for the message redelivery backoff [pulsar]

2024-06-05 Thread via GitHub 


RobertIndie opened a new pull request, #22855:
URL: https://github.com/apache/pulsar/pull/22855

   
   
   ## Motivation
   
   The document states that `ackTimeoutRedeliveryBackoff` cannot be used with 
`consumer.negativeAcknowledge(MessageId messageId)`. However, this is 
confusing. The `ackTimeoutRedeliveryBackoff` should not relate to the nack.
   
   ## Modification
   
   - Fix the doc for the `ackTimeoutRedeliveryBackoff`
   - Improve the doc for `negativeAckRedeliveryBackoff` and 
`ackTimeoutRedeliveryBackoff`
   
   ### Does this pull request potentially affect one of the following parts:
   
   
   
   *If the box was checked, please highlight the changes*
   
   - [ ] Dependencies (add or upgrade a dependency)
   - [ ] The public API
   - [ ] The schema
   - [ ] The default values of configurations
   - [ ] The threading model
   - [ ] The binary protocol
   - [ ] The REST endpoints
   - [ ] The admin CLI options
   - [ ] The metrics
   - [ ] Anything that affects deployment
   
   ### Documentation
   
   
   
   - [x] `doc` 
   - [ ] `doc-required` 
   - [ ] `doc-not-needed` 
   - [ ] `doc-complete` 
   
   ### Matching PR in forked repository
   
   PR in forked repository: 
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Add ack timeout support [pulsar-client-go]

2024-06-05 Thread via GitHub 


RobertIndie commented on issue #403:
URL: 
https://github.com/apache/pulsar-client-go/issues/403#issuecomment-2151354334

   > I was talking about this Ack Timeout in the Java Consumer
   
https://github.com/codelipenghui/incubator-pulsar/blob/a7f9670402a54b99fbc7ac9717a837092cf444cf/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerBuilderImpl.java#L63
   
   I think this configuration has been deprecated in recent versions. And yes. 
The golang client is different. The golang doesn't have the ack timeout setting.
   
   > if my various paths through the code don't result in an Ack or Nack
   will the message be redelivered to existing consumers - it sounds like the 
answer is no.
   
   No. You should either use `Nack` or `ReconsumeLater` to redelivery those 
messages.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

svn commit: r69556 - /dev/pulsar/pulsar-3.3.0-candidate-4/ /release/pulsar/pulsar-3.3.0/

2024-06-05 Thread penghui 
Author: penghui
Date: Thu Jun  6 02:11:01 2024
New Revision: 69556

Log:
Release Apache Pulsar 3.3.0

Added:
release/pulsar/pulsar-3.3.0/
  - copied from r69555, dev/pulsar/pulsar-3.3.0-candidate-4/
Removed:
dev/pulsar/pulsar-3.3.0-candidate-4/

(pulsar-site) branch main updated: Docs sync done from apache/pulsar (#d74010c)

2024-06-05 Thread urfree 
This is an automated email from the ASF dual-hosted git repository.

urfree pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/pulsar-site.git


The following commit(s) were added to refs/heads/main by this push:
 new 4eee778c58e6 Docs sync done from apache/pulsar (#d74010c)
4eee778c58e6 is described below

commit 4eee778c58e6c1ca6c8b1c37aa03876d20dd2ba4
Author: github-actions[bot] 
<41898282+github-actions[bot]@users.noreply.github.com>
AuthorDate: Thu Jun 6 01:36:17 2024 +

Docs sync done from apache/pulsar (#d74010c)
---
 static/reference/next/config/reference-configuration-broker.md | 2 +-
 static/reference/next/config/reference-configuration-standalone.md | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/static/reference/next/config/reference-configuration-broker.md 
b/static/reference/next/config/reference-configuration-broker.md
index b18f07efd887..4165194386c4 100644
--- a/static/reference/next/config/reference-configuration-broker.md
+++ b/static/reference/next/config/reference-configuration-broker.md
@@ -5528,7 +5528,7 @@ Memory Resource Usage Weight. Deprecated: Memory is no 
longer used as a load bal
 
 **Type**: `double`
 
-**Default**: `1.0`
+**Default**: `0.0`
 
 **Dynamic**: `true`
 
diff --git a/static/reference/next/config/reference-configuration-standalone.md 
b/static/reference/next/config/reference-configuration-standalone.md
index b18f07efd887..4165194386c4 100644
--- a/static/reference/next/config/reference-configuration-standalone.md
+++ b/static/reference/next/config/reference-configuration-standalone.md
@@ -5528,7 +5528,7 @@ Memory Resource Usage Weight. Deprecated: Memory is no 
longer used as a load bal
 
 **Type**: `double`
 
-**Default**: `1.0`
+**Default**: `0.0`
 
 **Dynamic**: `true`

Re: [PR] [improve] Refactored BK ClientFactory to return futures [pulsar]

2024-06-05 Thread via GitHub 


merlimat commented on PR #22853:
URL: https://github.com/apache/pulsar/pull/22853#issuecomment-2151172505

   I'm skipping backporting to `branch-3.0` because there are several conflicts.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(pulsar) 02/02: [improve] Refactored BK ClientFactory to return futures (#22853)

2024-06-05 Thread mmerli 
This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit d97314458eb58afd7e70f5292d775a7f073ccbc1
Author: Matteo Merli 
AuthorDate: Wed Jun 5 17:09:32 2024 -0700

[improve] Refactored BK ClientFactory to return futures (#22853)
---
 .../mledger/impl/ManagedLedgerFactoryImpl.java | 223 ++---
 .../mledger/impl/ManagedLedgerOfflineBacklog.java  |  20 +-
 .../pulsar/broker/BookKeeperClientFactory.java |  19 +-
 .../pulsar/broker/BookKeeperClientFactoryImpl.java |  28 +--
 .../pulsar/broker/ManagedLedgerClientFactory.java  |  39 ++--
 .../bucket/BookkeeperBucketSnapshotStorage.java|   2 +-
 .../service/schema/BookkeeperSchemaStorage.java|   2 +-
 .../apache/pulsar/compaction/CompactorTool.java|   2 +-
 .../broker/MockedBookKeeperClientFactory.java  |  18 +-
 .../testcontext/MockBookKeeperClientFactory.java   |  15 +-
 .../pulsar/compaction/CompactedTopicTest.java  |   6 +-
 .../pulsar/compaction/CompactionRetentionTest.java |   2 +-
 .../apache/pulsar/compaction/CompactionTest.java   |   2 +-
 .../apache/pulsar/compaction/CompactorTest.java|   2 +-
 .../compaction/ServiceUnitStateCompactionTest.java |   2 +-
 .../compaction/TopicCompactionServiceTest.java |   2 +-
 16 files changed, 193 insertions(+), 191 deletions(-)

diff --git 
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryImpl.java
 
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryImpl.java
index d867f2f4c02..ed803a81462 100644
--- 
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryImpl.java
+++ 
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryImpl.java
@@ -161,7 +161,7 @@ public class ManagedLedgerFactoryImpl implements 
ManagedLedgerFactory {
 public ManagedLedgerFactoryImpl(MetadataStoreExtended metadataStore, 
BookKeeper bookKeeper,
 ManagedLedgerFactoryConfig config)
 throws Exception {
-this(metadataStore, (policyConfig) -> bookKeeper, config);
+this(metadataStore, (policyConfig) -> 
CompletableFuture.completedFuture(bookKeeper), config);
 }
 
 public ManagedLedgerFactoryImpl(MetadataStoreExtended metadataStore,
@@ -233,8 +233,8 @@ public class ManagedLedgerFactoryImpl implements 
ManagedLedgerFactory {
 }
 
 @Override
-public BookKeeper get(EnsemblePlacementPolicyConfig policy) {
-return bkClient;
+public CompletableFuture get(EnsemblePlacementPolicyConfig 
policy) {
+return CompletableFuture.completedFuture(bkClient);
 }
 }
 
@@ -378,56 +378,63 @@ public class ManagedLedgerFactoryImpl implements 
ManagedLedgerFactory {
 ledgers.computeIfAbsent(name, (mlName) -> {
 // Create the managed ledger
 CompletableFuture future = new 
CompletableFuture<>();
-BookKeeper bk = bookkeeperFactory.get(
-new 
EnsemblePlacementPolicyConfig(config.getBookKeeperEnsemblePlacementPolicyClassName(),
-
config.getBookKeeperEnsemblePlacementPolicyProperties()));
-final ManagedLedgerImpl newledger = config.getShadowSource() == 
null
-? new ManagedLedgerImpl(this, bk, store, config, 
scheduledExecutor, name, mlOwnershipChecker)
-: new ShadowManagedLedgerImpl(this, bk, store, config, 
scheduledExecutor, name,
-mlOwnershipChecker);
-PendingInitializeManagedLedger pendingLedger = new 
PendingInitializeManagedLedger(newledger);
-pendingInitializeLedgers.put(name, pendingLedger);
-newledger.initialize(new ManagedLedgerInitializeLedgerCallback() {
-@Override
-public void initializeComplete() {
-log.info("[{}] Successfully initialize managed ledger", 
name);
-pendingInitializeLedgers.remove(name, pendingLedger);
-future.complete(newledger);
-
-// May need to update the cursor position
-newledger.maybeUpdateCursorBeforeTrimmingConsumedLedger();
-// May need to trigger offloading
-if (config.isTriggerOffloadOnTopicLoad()) {
-
newledger.maybeOffloadInBackground(NULL_OFFLOAD_PROMISE);
-}
-}
-
-@Override
-public void initializeFailed(ManagedLedgerException e) {
-if (config.isCreateIfMissing()) {
-log.error("[{}] Failed to initialize managed ledger: 
{}", name, e.getMessage());
-}
-
-// Clean the map if initialization fails
-ledgers.remove(name, future);
-
-

(pulsar) 01/02: [fix] Remove blocking calls from BookieRackAffinityMapping (#22846)

2024-06-05 Thread mmerli 
This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit 190857e1854271b55ba84d63422a473ae45e64fc
Author: Matteo Merli 
AuthorDate: Wed Jun 5 10:49:00 2024 -0700

[fix] Remove blocking calls from BookieRackAffinityMapping (#22846)
---
 .../rackawareness/BookieRackAffinityMapping.java   | 44 +-
 .../IsolatedBookieEnsemblePlacementPolicy.java |  2 +-
 2 files changed, 28 insertions(+), 18 deletions(-)

diff --git 
a/pulsar-broker-common/src/main/java/org/apache/pulsar/bookie/rackawareness/BookieRackAffinityMapping.java
 
b/pulsar-broker-common/src/main/java/org/apache/pulsar/bookie/rackawareness/BookieRackAffinityMapping.java
index 983822f2294..4a5ff746f40 100644
--- 
a/pulsar-broker-common/src/main/java/org/apache/pulsar/bookie/rackawareness/BookieRackAffinityMapping.java
+++ 
b/pulsar-broker-common/src/main/java/org/apache/pulsar/bookie/rackawareness/BookieRackAffinityMapping.java
@@ -70,7 +70,7 @@ public class BookieRackAffinityMapping extends 
AbstractDNSToSwitchMapping
 private BookiesRackConfiguration racksWithHost = new 
BookiesRackConfiguration();
 private Map bookieInfoMap = new HashMap<>();
 
-public static MetadataStore createMetadataStore(Configuration conf) throws 
MetadataException {
+static MetadataStore getMetadataStore(Configuration conf) throws 
MetadataException {
 MetadataStore store;
 Object storeProperty = conf.getProperty(METADATA_STORE_INSTANCE);
 if (storeProperty != null) {
@@ -116,12 +116,20 @@ public class BookieRackAffinityMapping extends 
AbstractDNSToSwitchMapping
 super.setConf(conf);
 MetadataStore store;
 try {
-store = createMetadataStore(conf);
-bookieMappingCache = 
store.getMetadataCache(BookiesRackConfiguration.class);
-store.registerListener(this::handleUpdates);
-racksWithHost = bookieMappingCache.get(BOOKIE_INFO_ROOT_PATH).get()
-.orElseGet(BookiesRackConfiguration::new);
-for (Map bookieMapping : 
racksWithHost.values()) {
+store = getMetadataStore(conf);
+} catch (MetadataException e) {
+throw new RuntimeException(METADATA_STORE_INSTANCE + " failed to 
init BookieId list");
+}
+
+bookieMappingCache = 
store.getMetadataCache(BookiesRackConfiguration.class);
+store.registerListener(this::handleUpdates);
+
+try {
+var racksWithHost = bookieMappingCache.get(BOOKIE_INFO_ROOT_PATH)
+.thenApply(optRes -> 
optRes.orElseGet(BookiesRackConfiguration::new))
+.get();
+
+for (var bookieMapping : racksWithHost.values()) {
 for (String address : bookieMapping.keySet()) {
 bookieAddressListLastTime.add(BookieId.parse(address));
 }
@@ -131,10 +139,12 @@ public class BookieRackAffinityMapping extends 
AbstractDNSToSwitchMapping
 }
 }
 updateRacksWithHost(racksWithHost);
-watchAvailableBookies();
-} catch (InterruptedException | ExecutionException | MetadataException 
e) {
-throw new RuntimeException(METADATA_STORE_INSTANCE + " failed to 
init BookieId list");
+} catch (ExecutionException | InterruptedException e) {
+LOG.error("Failed to update rack info. ", e);
+throw new RuntimeException(e);
 }
+
+watchAvailableBookies();
 }
 
 private void watchAvailableBookies() {
@@ -145,13 +155,13 @@ public class BookieRackAffinityMapping extends 
AbstractDNSToSwitchMapping
 field.setAccessible(true);
 RegistrationClient registrationClient = (RegistrationClient) 
field.get(bookieAddressResolver);
 registrationClient.watchWritableBookies(versioned -> {
-try {
-racksWithHost = 
bookieMappingCache.get(BOOKIE_INFO_ROOT_PATH).get()
-.orElseGet(BookiesRackConfiguration::new);
-updateRacksWithHost(racksWithHost);
-} catch (InterruptedException | ExecutionException e) {
-LOG.error("Failed to update rack info. ", e);
-}
+bookieMappingCache.get(BOOKIE_INFO_ROOT_PATH)
+.thenApply(optRes -> 
optRes.orElseGet(BookiesRackConfiguration::new))
+.thenAccept(this::updateRacksWithHost)
+.exceptionally(ex -> {
+LOG.error("Failed to update rack info. ", ex);
+return null;
+});
 });
 } catch (NoSuchFieldException | IllegalAccessException e) {
 LOG.error("Failed watch available

(pulsar) branch branch-3.2 updated (febb50cc13a -> d97314458eb)

2024-06-05 Thread mmerli 
This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a change to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git


from febb50cc13a [improve] Upgrade Jetcd to 0.7.7 and VertX to 4.5.8 
(#22835)
 new 190857e1854 [fix] Remove blocking calls from BookieRackAffinityMapping 
(#22846)
 new d97314458eb [improve] Refactored BK ClientFactory to return futures 
(#22853)

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../mledger/impl/ManagedLedgerFactoryImpl.java | 223 ++---
 .../mledger/impl/ManagedLedgerOfflineBacklog.java  |  20 +-
 .../rackawareness/BookieRackAffinityMapping.java   |  44 ++--
 .../IsolatedBookieEnsemblePlacementPolicy.java |   2 +-
 .../pulsar/broker/BookKeeperClientFactory.java |  19 +-
 .../pulsar/broker/BookKeeperClientFactoryImpl.java |  28 +--
 .../pulsar/broker/ManagedLedgerClientFactory.java  |  39 ++--
 .../bucket/BookkeeperBucketSnapshotStorage.java|   2 +-
 .../service/schema/BookkeeperSchemaStorage.java|   2 +-
 .../apache/pulsar/compaction/CompactorTool.java|   2 +-
 .../broker/MockedBookKeeperClientFactory.java  |  18 +-
 .../testcontext/MockBookKeeperClientFactory.java   |  15 +-
 .../pulsar/compaction/CompactedTopicTest.java  |   6 +-
 .../pulsar/compaction/CompactionRetentionTest.java |   2 +-
 .../apache/pulsar/compaction/CompactionTest.java   |   2 +-
 .../apache/pulsar/compaction/CompactorTest.java|   2 +-
 .../compaction/ServiceUnitStateCompactionTest.java |   2 +-
 .../compaction/TopicCompactionServiceTest.java |   2 +-
 18 files changed, 221 insertions(+), 209 deletions(-)

(pulsar) 01/02: [fix] Remove blocking calls from BookieRackAffinityMapping (#22846)

2024-06-05 Thread mmerli 
This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit 3e2ca291d3e32d71caa60e5b3ee0be9b920a0626
Author: Matteo Merli 
AuthorDate: Wed Jun 5 10:49:00 2024 -0700

[fix] Remove blocking calls from BookieRackAffinityMapping (#22846)
---
 .../rackawareness/BookieRackAffinityMapping.java   | 44 +-
 .../IsolatedBookieEnsemblePlacementPolicy.java |  2 +-
 2 files changed, 28 insertions(+), 18 deletions(-)

diff --git 
a/pulsar-broker-common/src/main/java/org/apache/pulsar/bookie/rackawareness/BookieRackAffinityMapping.java
 
b/pulsar-broker-common/src/main/java/org/apache/pulsar/bookie/rackawareness/BookieRackAffinityMapping.java
index 983822f2294..4a5ff746f40 100644
--- 
a/pulsar-broker-common/src/main/java/org/apache/pulsar/bookie/rackawareness/BookieRackAffinityMapping.java
+++ 
b/pulsar-broker-common/src/main/java/org/apache/pulsar/bookie/rackawareness/BookieRackAffinityMapping.java
@@ -70,7 +70,7 @@ public class BookieRackAffinityMapping extends 
AbstractDNSToSwitchMapping
 private BookiesRackConfiguration racksWithHost = new 
BookiesRackConfiguration();
 private Map bookieInfoMap = new HashMap<>();
 
-public static MetadataStore createMetadataStore(Configuration conf) throws 
MetadataException {
+static MetadataStore getMetadataStore(Configuration conf) throws 
MetadataException {
 MetadataStore store;
 Object storeProperty = conf.getProperty(METADATA_STORE_INSTANCE);
 if (storeProperty != null) {
@@ -116,12 +116,20 @@ public class BookieRackAffinityMapping extends 
AbstractDNSToSwitchMapping
 super.setConf(conf);
 MetadataStore store;
 try {
-store = createMetadataStore(conf);
-bookieMappingCache = 
store.getMetadataCache(BookiesRackConfiguration.class);
-store.registerListener(this::handleUpdates);
-racksWithHost = bookieMappingCache.get(BOOKIE_INFO_ROOT_PATH).get()
-.orElseGet(BookiesRackConfiguration::new);
-for (Map bookieMapping : 
racksWithHost.values()) {
+store = getMetadataStore(conf);
+} catch (MetadataException e) {
+throw new RuntimeException(METADATA_STORE_INSTANCE + " failed to 
init BookieId list");
+}
+
+bookieMappingCache = 
store.getMetadataCache(BookiesRackConfiguration.class);
+store.registerListener(this::handleUpdates);
+
+try {
+var racksWithHost = bookieMappingCache.get(BOOKIE_INFO_ROOT_PATH)
+.thenApply(optRes -> 
optRes.orElseGet(BookiesRackConfiguration::new))
+.get();
+
+for (var bookieMapping : racksWithHost.values()) {
 for (String address : bookieMapping.keySet()) {
 bookieAddressListLastTime.add(BookieId.parse(address));
 }
@@ -131,10 +139,12 @@ public class BookieRackAffinityMapping extends 
AbstractDNSToSwitchMapping
 }
 }
 updateRacksWithHost(racksWithHost);
-watchAvailableBookies();
-} catch (InterruptedException | ExecutionException | MetadataException 
e) {
-throw new RuntimeException(METADATA_STORE_INSTANCE + " failed to 
init BookieId list");
+} catch (ExecutionException | InterruptedException e) {
+LOG.error("Failed to update rack info. ", e);
+throw new RuntimeException(e);
 }
+
+watchAvailableBookies();
 }
 
 private void watchAvailableBookies() {
@@ -145,13 +155,13 @@ public class BookieRackAffinityMapping extends 
AbstractDNSToSwitchMapping
 field.setAccessible(true);
 RegistrationClient registrationClient = (RegistrationClient) 
field.get(bookieAddressResolver);
 registrationClient.watchWritableBookies(versioned -> {
-try {
-racksWithHost = 
bookieMappingCache.get(BOOKIE_INFO_ROOT_PATH).get()
-.orElseGet(BookiesRackConfiguration::new);
-updateRacksWithHost(racksWithHost);
-} catch (InterruptedException | ExecutionException e) {
-LOG.error("Failed to update rack info. ", e);
-}
+bookieMappingCache.get(BOOKIE_INFO_ROOT_PATH)
+.thenApply(optRes -> 
optRes.orElseGet(BookiesRackConfiguration::new))
+.thenAccept(this::updateRacksWithHost)
+.exceptionally(ex -> {
+LOG.error("Failed to update rack info. ", ex);
+return null;
+});
 });
 } catch (NoSuchFieldException | IllegalAccessException e) {
 LOG.error("Failed watch available

(pulsar) 02/02: [improve] Refactored BK ClientFactory to return futures (#22853)

2024-06-05 Thread mmerli 
This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit 217f1f011b66677d6d78ff9d42837c864ae10104
Author: Matteo Merli 
AuthorDate: Wed Jun 5 17:09:32 2024 -0700

[improve] Refactored BK ClientFactory to return futures (#22853)
---
 .../mledger/impl/ManagedLedgerFactoryImpl.java | 213 ++---
 .../mledger/impl/ManagedLedgerOfflineBacklog.java  |  20 +-
 .../pulsar/broker/BookKeeperClientFactory.java |  19 +-
 .../pulsar/broker/BookKeeperClientFactoryImpl.java |  28 +--
 .../pulsar/broker/ManagedLedgerClientFactory.java  |  39 ++--
 .../bucket/BookkeeperBucketSnapshotStorage.java|   2 +-
 .../service/schema/BookkeeperSchemaStorage.java|   2 +-
 .../apache/pulsar/compaction/CompactorTool.java|   2 +-
 .../broker/MockedBookKeeperClientFactory.java  |  18 +-
 .../testcontext/MockBookKeeperClientFactory.java   |  15 +-
 .../pulsar/compaction/CompactedTopicTest.java  |   6 +-
 .../pulsar/compaction/CompactionRetentionTest.java |   2 +-
 .../apache/pulsar/compaction/CompactionTest.java   |   2 +-
 .../apache/pulsar/compaction/CompactorTest.java|   2 +-
 .../compaction/ServiceUnitStateCompactionTest.java |   2 +-
 .../compaction/TopicCompactionServiceTest.java |   2 +-
 16 files changed, 188 insertions(+), 186 deletions(-)

diff --git 
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryImpl.java
 
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryImpl.java
index 5ce84b3ed85..a0929044a6a 100644
--- 
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryImpl.java
+++ 
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryImpl.java
@@ -160,7 +160,7 @@ public class ManagedLedgerFactoryImpl implements 
ManagedLedgerFactory {
 public ManagedLedgerFactoryImpl(MetadataStoreExtended metadataStore, 
BookKeeper bookKeeper,
 ManagedLedgerFactoryConfig config)
 throws Exception {
-this(metadataStore, (policyConfig) -> bookKeeper, config);
+this(metadataStore, (policyConfig) -> 
CompletableFuture.completedFuture(bookKeeper), config);
 }
 
 public ManagedLedgerFactoryImpl(MetadataStoreExtended metadataStore,
@@ -232,8 +232,8 @@ public class ManagedLedgerFactoryImpl implements 
ManagedLedgerFactory {
 }
 
 @Override
-public BookKeeper get(EnsemblePlacementPolicyConfig policy) {
-return bkClient;
+public CompletableFuture get(EnsemblePlacementPolicyConfig 
policy) {
+return CompletableFuture.completedFuture(bkClient);
 }
 }
 
@@ -377,52 +377,59 @@ public class ManagedLedgerFactoryImpl implements 
ManagedLedgerFactory {
 ledgers.computeIfAbsent(name, (mlName) -> {
 // Create the managed ledger
 CompletableFuture future = new 
CompletableFuture<>();
-BookKeeper bk = bookkeeperFactory.get(
-new 
EnsemblePlacementPolicyConfig(config.getBookKeeperEnsemblePlacementPolicyClassName(),
-
config.getBookKeeperEnsemblePlacementPolicyProperties()));
-final ManagedLedgerImpl newledger = config.getShadowSource() == 
null
-? new ManagedLedgerImpl(this, bk, store, config, 
scheduledExecutor, name, mlOwnershipChecker)
-: new ShadowManagedLedgerImpl(this, bk, store, config, 
scheduledExecutor, name,
-mlOwnershipChecker);
-PendingInitializeManagedLedger pendingLedger = new 
PendingInitializeManagedLedger(newledger);
-pendingInitializeLedgers.put(name, pendingLedger);
-newledger.initialize(new ManagedLedgerInitializeLedgerCallback() {
-@Override
-public void initializeComplete() {
-log.info("[{}] Successfully initialize managed ledger", 
name);
-pendingInitializeLedgers.remove(name, pendingLedger);
-future.complete(newledger);
+bookkeeperFactory.get(
+new 
EnsemblePlacementPolicyConfig(config.getBookKeeperEnsemblePlacementPolicyClassName(),
+
config.getBookKeeperEnsemblePlacementPolicyProperties()))
+.thenAccept(bk -> {
+final ManagedLedgerImpl newledger = 
config.getShadowSource() == null
+? new ManagedLedgerImpl(this, bk, store, 
config, scheduledExecutor, name,
+mlOwnershipChecker)
+: new ShadowManagedLedgerImpl(this, bk, store, 
config, scheduledExecutor, name,
+mlOwnershipChecker);
+PendingInitializeManagedLedger pendingLedger = new

(pulsar) branch branch-3.3 updated (abe6d79510c -> 217f1f011b6)

2024-06-05 Thread mmerli 
This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a change to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/pulsar.git


from abe6d79510c [fix][broker] Fix ProducerBusy issue due to incorrect 
userCreatedProducerCount on non-persistent topic (#22685)
 new 3e2ca291d3e [fix] Remove blocking calls from BookieRackAffinityMapping 
(#22846)
 new 217f1f011b6 [improve] Refactored BK ClientFactory to return futures 
(#22853)

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../mledger/impl/ManagedLedgerFactoryImpl.java | 213 ++---
 .../mledger/impl/ManagedLedgerOfflineBacklog.java  |  20 +-
 .../rackawareness/BookieRackAffinityMapping.java   |  44 +++--
 .../IsolatedBookieEnsemblePlacementPolicy.java |   2 +-
 .../pulsar/broker/BookKeeperClientFactory.java |  19 +-
 .../pulsar/broker/BookKeeperClientFactoryImpl.java |  28 +--
 .../pulsar/broker/ManagedLedgerClientFactory.java  |  39 ++--
 .../bucket/BookkeeperBucketSnapshotStorage.java|   2 +-
 .../service/schema/BookkeeperSchemaStorage.java|   2 +-
 .../apache/pulsar/compaction/CompactorTool.java|   2 +-
 .../broker/MockedBookKeeperClientFactory.java  |  18 +-
 .../testcontext/MockBookKeeperClientFactory.java   |  15 +-
 .../pulsar/compaction/CompactedTopicTest.java  |   6 +-
 .../pulsar/compaction/CompactionRetentionTest.java |   2 +-
 .../apache/pulsar/compaction/CompactionTest.java   |   2 +-
 .../apache/pulsar/compaction/CompactorTest.java|   2 +-
 .../compaction/ServiceUnitStateCompactionTest.java |   2 +-
 .../compaction/TopicCompactionServiceTest.java |   2 +-
 18 files changed, 216 insertions(+), 204 deletions(-)

Re: [I] [Bug] Deadlock in broker service while initializing bkClient [pulsar]

2024-06-05 Thread via GitHub 


merlimat closed issue #22699: [Bug] Deadlock in broker service while 
initializing bkClient
URL: https://github.com/apache/pulsar/issues/22699


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(pulsar) branch master updated: [improve] Refactored BK ClientFactory to return futures (#22853)

2024-06-05 Thread mmerli 
This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new d74010c271a [improve] Refactored BK ClientFactory to return futures 
(#22853)
d74010c271a is described below

commit d74010c271abfb0a77a4dacf0ab072a957afeb5a
Author: Matteo Merli 
AuthorDate: Wed Jun 5 17:09:32 2024 -0700

[improve] Refactored BK ClientFactory to return futures (#22853)
---
 .../mledger/impl/ManagedLedgerFactoryImpl.java | 223 ++---
 .../mledger/impl/ManagedLedgerOfflineBacklog.java  |  20 +-
 .../pulsar/broker/BookKeeperClientFactory.java |  19 +-
 .../pulsar/broker/BookKeeperClientFactoryImpl.java |  28 +--
 .../pulsar/broker/ManagedLedgerClientFactory.java  |  39 ++--
 .../bucket/BookkeeperBucketSnapshotStorage.java|   2 +-
 .../service/schema/BookkeeperSchemaStorage.java|   2 +-
 .../apache/pulsar/compaction/CompactorTool.java|   2 +-
 .../broker/MockedBookKeeperClientFactory.java  |  18 +-
 .../testcontext/MockBookKeeperClientFactory.java   |  15 +-
 .../pulsar/compaction/CompactedTopicTest.java  |   6 +-
 .../pulsar/compaction/CompactionRetentionTest.java |   2 +-
 .../apache/pulsar/compaction/CompactionTest.java   |   2 +-
 .../apache/pulsar/compaction/CompactorTest.java|   2 +-
 .../compaction/ServiceUnitStateCompactionTest.java |   2 +-
 .../compaction/TopicCompactionServiceTest.java |   2 +-
 16 files changed, 193 insertions(+), 191 deletions(-)

diff --git 
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryImpl.java
 
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryImpl.java
index d867f2f4c02..ed803a81462 100644
--- 
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryImpl.java
+++ 
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryImpl.java
@@ -161,7 +161,7 @@ public class ManagedLedgerFactoryImpl implements 
ManagedLedgerFactory {
 public ManagedLedgerFactoryImpl(MetadataStoreExtended metadataStore, 
BookKeeper bookKeeper,
 ManagedLedgerFactoryConfig config)
 throws Exception {
-this(metadataStore, (policyConfig) -> bookKeeper, config);
+this(metadataStore, (policyConfig) -> 
CompletableFuture.completedFuture(bookKeeper), config);
 }
 
 public ManagedLedgerFactoryImpl(MetadataStoreExtended metadataStore,
@@ -233,8 +233,8 @@ public class ManagedLedgerFactoryImpl implements 
ManagedLedgerFactory {
 }
 
 @Override
-public BookKeeper get(EnsemblePlacementPolicyConfig policy) {
-return bkClient;
+public CompletableFuture get(EnsemblePlacementPolicyConfig 
policy) {
+return CompletableFuture.completedFuture(bkClient);
 }
 }
 
@@ -378,56 +378,63 @@ public class ManagedLedgerFactoryImpl implements 
ManagedLedgerFactory {
 ledgers.computeIfAbsent(name, (mlName) -> {
 // Create the managed ledger
 CompletableFuture future = new 
CompletableFuture<>();
-BookKeeper bk = bookkeeperFactory.get(
-new 
EnsemblePlacementPolicyConfig(config.getBookKeeperEnsemblePlacementPolicyClassName(),
-
config.getBookKeeperEnsemblePlacementPolicyProperties()));
-final ManagedLedgerImpl newledger = config.getShadowSource() == 
null
-? new ManagedLedgerImpl(this, bk, store, config, 
scheduledExecutor, name, mlOwnershipChecker)
-: new ShadowManagedLedgerImpl(this, bk, store, config, 
scheduledExecutor, name,
-mlOwnershipChecker);
-PendingInitializeManagedLedger pendingLedger = new 
PendingInitializeManagedLedger(newledger);
-pendingInitializeLedgers.put(name, pendingLedger);
-newledger.initialize(new ManagedLedgerInitializeLedgerCallback() {
-@Override
-public void initializeComplete() {
-log.info("[{}] Successfully initialize managed ledger", 
name);
-pendingInitializeLedgers.remove(name, pendingLedger);
-future.complete(newledger);
-
-// May need to update the cursor position
-newledger.maybeUpdateCursorBeforeTrimmingConsumedLedger();
-// May need to trigger offloading
-if (config.isTriggerOffloadOnTopicLoad()) {
-
newledger.maybeOffloadInBackground(NULL_OFFLOAD_PROMISE);
-}
-}
-
-@Override
-public void initializeFailed(ManagedLedgerException e) {
-if (config.isCreateIfMissing()) {
-log.error("[{}] Failed to initialize

Re: [PR] [improve] Refactored BK ClientFactory to return futures [pulsar]

2024-06-05 Thread via GitHub 


merlimat merged PR #22853:
URL: https://github.com/apache/pulsar/pull/22853


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [feat][broker] PIP-264: Add broker web executor metrics [pulsar]

2024-06-05 Thread via GitHub 


merlimat merged PR #22816:
URL: https://github.com/apache/pulsar/pull/22816


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(pulsar) branch master updated: [feat][broker] PIP-264: Add broker web executor metrics (#22816)

2024-06-05 Thread mmerli 
This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 4341f0f301e [feat][broker] PIP-264: Add broker web executor metrics 
(#22816)
4341f0f301e is described below

commit 4341f0f301e0da344bb5ce07bc62c373e7ce48ef
Author: Dragos Misca 
AuthorDate: Wed Jun 5 16:34:56 2024 -0700

[feat][broker] PIP-264: Add broker web executor metrics (#22816)
---
 .../broker/web/WebExecutorThreadPoolStats.java | 83 ++
 .../apache/pulsar/broker/web/WebExecutorStats.java |  7 ++
 .../org/apache/pulsar/broker/web/WebService.java   |  5 ++
 .../apache/pulsar/broker/web/WebServiceTest.java   | 18 +
 4 files changed, 113 insertions(+)

diff --git 
a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/web/WebExecutorThreadPoolStats.java
 
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/web/WebExecutorThreadPoolStats.java
new file mode 100644
index 000..6bfe4e33b8e
--- /dev/null
+++ 
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/web/WebExecutorThreadPoolStats.java
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.web;
+
+import com.google.common.annotations.VisibleForTesting;
+import io.opentelemetry.api.common.AttributeKey;
+import io.opentelemetry.api.common.Attributes;
+import io.opentelemetry.api.metrics.Meter;
+import io.opentelemetry.api.metrics.ObservableLongUpDownCounter;
+
+public class WebExecutorThreadPoolStats implements AutoCloseable {
+// Replaces ['pulsar_web_executor_max_threads', 
'pulsar_web_executor_min_threads']
+public static final String LIMIT_COUNTER = 
"pulsar.web.executor.thread.limit";
+private final ObservableLongUpDownCounter limitCounter;
+
+// Replaces
+// ['pulsar_web_executor_active_threads', 
'pulsar_web_executor_current_threads', 'pulsar_web_executor_idle_threads']
+public static final String USAGE_COUNTER = 
"pulsar.web.executor.thread.usage";
+private final ObservableLongUpDownCounter usageCounter;
+
+public static final AttributeKey LIMIT_TYPE_KEY =
+AttributeKey.stringKey("pulsar.web.executor.thread.limit.type");
+@VisibleForTesting
+enum LimitType {
+MAX,
+MIN;
+public final Attributes attributes = Attributes.of(LIMIT_TYPE_KEY, 
name().toLowerCase());
+}
+
+public static final AttributeKey USAGE_TYPE_KEY =
+AttributeKey.stringKey("pulsar.web.executor.thread.usage.type");
+@VisibleForTesting
+enum UsageType {
+ACTIVE,
+CURRENT,
+IDLE;
+public final Attributes attributes = Attributes.of(USAGE_TYPE_KEY, 
name().toLowerCase());
+}
+
+public WebExecutorThreadPoolStats(Meter meter, WebExecutorThreadPool 
executor) {
+limitCounter = meter
+.upDownCounterBuilder(LIMIT_COUNTER)
+.setUnit("{thread}")
+.setDescription("The thread limits for the pulsar-web executor 
pool.")
+.buildWithCallback(measurement -> {
+measurement.record(executor.getMaxThreads(), 
LimitType.MAX.attributes);
+measurement.record(executor.getMinThreads(), 
LimitType.MIN.attributes);
+});
+usageCounter = meter
+.upDownCounterBuilder(USAGE_COUNTER)
+.setUnit("{thread}")
+.setDescription("The current usage of threads in the 
pulsar-web executor pool.")
+.buildWithCallback(measurement -> {
+var idleThreads = executor.getIdleThreads();
+var currentThreads = executor.getThreads();
+measurement.record(idleThreads, UsageType.IDLE.attributes);
+measurement.record(currentThreads, 
UsageType.CURRENT.attributes);
+measurement.record(currentThreads - idleThreads, 
UsageType.ACTIVE.attributes);
+});
+}
+
+@Override
+public synchronized void close() {
+limitCounter.close();
+

Re: [PR] [improve] Refactored BK ClientFactory to return futures [pulsar]

2024-06-05 Thread via GitHub 


rdhabalia commented on PR #22853:
URL: https://github.com/apache/pulsar/pull/22853#issuecomment-2151047012

   > I honestly don't understand how can you be polemic about this. It baffles 
me.
   
   I said what I saw. I have closed all other PRs and we can merge this one.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [fix] [broker] Fix broker deadlock to add timeout on blocking call during bk-client creation [pulsar]

2024-06-05 Thread via GitHub 


rdhabalia closed pull request #22843: [fix] [broker] Fix broker deadlock to add 
timeout on blocking call during bk-client creation
URL: https://github.com/apache/pulsar/pull/22843


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [fix] [broker] Fix broker deadlock by making blocking bk-create async during ml-creation [pulsar]

2024-06-05 Thread via GitHub 


rdhabalia closed pull request #22842: [fix] [broker] Fix broker deadlock by 
making blocking bk-create async during ml-creation
URL: https://github.com/apache/pulsar/pull/22842


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [fix] [broker] Fix broker deadlock in metadata-callback thread due to single thread processing [pulsar]

2024-06-05 Thread via GitHub 


rdhabalia closed pull request #22841: [fix] [broker] Fix broker deadlock in 
metadata-callback thread due to single thread processing
URL: https://github.com/apache/pulsar/pull/22841


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] [Bug] Broker became irresponsive due to deadlock during race-condition in metadatastore callback [pulsar]

2024-06-05 Thread via GitHub 


rdhabalia closed issue #22840: [Bug] Broker became irresponsive due to deadlock 
during race-condition in metadatastore callback
URL: https://github.com/apache/pulsar/issues/22840


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [improve] Refactored BK ClientFactory to return futures [pulsar]

2024-06-05 Thread via GitHub 


merlimat commented on PR #22853:
URL: https://github.com/apache/pulsar/pull/22853#issuecomment-2151024973

   > was not good where we ignore actual solution without proper review and 
other one was merged even if that approach was incorrect.
   
   ?!? It was so much ignored that you had 2 people jumping on your issue 
report within 2 hours, discussing multiple ways on how to best solve the issue.
   
   > I am not sure how this can help fo fix strict affinity and deadlock issue. 
 
   
   I think I've explained in the description and subsequent comment. It's 
similar to your approach, though in a more general way, it addresses all the 
usages where we are constructing the BK client, instead of just in one single 
spot.
   
   > and other one was merged even if that approach was incorrect. 
   
   I think you have not read my multiple comments on why that PR was needed.
   
   >  and please advise if anyone has plan to create PR so, one avoid doing 
duplicate efforts.
   
   ?? You posted your PR at the same time at the issue. I didn't fully agree 
with your proposed solution, so it's why I'm proposing a similar, though more 
general, way to fix the issue. 
   
   I honestly don't understand how can you be polemic about this. It baffles me.
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [improve] Refactored BK ClientFactory to return futures [pulsar]

2024-06-05 Thread via GitHub 


rdhabalia commented on PR #22853:
URL: https://github.com/apache/pulsar/pull/22853#issuecomment-2151011241

   I am not sure how this can help fo fix strict affinity and deadlock issue. 
also, sorry to say but what we did with #22846 and #22842 was not good where we 
ignore actual solution without proper review  and other one was merged even if 
that approach was incorrect. and please advise if anyone has plan to create PR 
so, one avoid doing duplicate efforts.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [improve][build] Support git worktree working directory while building docker images [pulsar]

2024-06-05 Thread via GitHub 


merlimat merged PR #22851:
URL: https://github.com/apache/pulsar/pull/22851


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(pulsar) branch master updated (74192871ed0 -> c23e677ae8c)

2024-06-05 Thread mmerli 
This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git


from 74192871ed0 [fix][meta] Check if metadata store is closed in 
RocksdbMetadataStore (#22852)
 add c23e677ae8c [improve][build] Support git worktree working directory 
while building docker images (#22851)

No new revisions were added by this update.

Summary of changes:
 docker/pulsar-all/pom.xml | 2 +-
 docker/pulsar/pom.xml | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

Re: [PR] [improve] Refactored BK ClientFactory to return futures [pulsar]

2024-06-05 Thread via GitHub 


merlimat commented on PR #22853:
URL: https://github.com/apache/pulsar/pull/22853#issuecomment-2151004493

   @rdhabalia 

   > if there is no issue with https://github.com/apache/pulsar/pull/22842 then 
can we merge it?
   
   There are still issues wiht #22842: 
   
1. It's only address 1 usage of a blocking API that is used in multiple 
places as if it were non-blocking
2. Does not solve the problem that if there's a timeout, it falls back to 
use default BK client, which might break the assumptions

   PTAL at this PR, it's a similar change, though it brings it closer to the BK 
client creation, so that all the factories expose non-blocking API. I believe 
it's a more general solution than #22842


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [fix] Remove blocking calls from BookieRackAffinityMapping [pulsar]

2024-06-05 Thread via GitHub 


merlimat commented on code in PR #22846:
URL: https://github.com/apache/pulsar/pull/22846#discussion_r1628470943


##
pulsar-broker-common/src/main/java/org/apache/pulsar/bookie/rackawareness/BookieRackAffinityMapping.java:
##
@@ -116,25 +115,32 @@ public synchronized void setConf(Configuration conf) {
 super.setConf(conf);
 MetadataStore store;
 try {
-store = createMetadataStore(conf);
-bookieMappingCache = 
store.getMetadataCache(BookiesRackConfiguration.class);
-store.registerListener(this::handleUpdates);
-racksWithHost = bookieMappingCache.get(BOOKIE_INFO_ROOT_PATH).get()
-.orElseGet(BookiesRackConfiguration::new);
-for (Map bookieMapping : 
racksWithHost.values()) {
-for (String address : bookieMapping.keySet()) {
-bookieAddressListLastTime.add(BookieId.parse(address));
-}
-if (LOG.isDebugEnabled()) {
-LOG.debug("BookieRackAffinityMapping init, 
bookieAddressListLastTime {}",
-bookieAddressListLastTime);
-}
-}
-updateRacksWithHost(racksWithHost);
-watchAvailableBookies();
-} catch (InterruptedException | ExecutionException | MetadataException 
e) {
+store = getMetadataStore(conf);
+} catch (MetadataException e) {
 throw new RuntimeException(METADATA_STORE_INSTANCE + " failed to 
init BookieId list");
 }
+
+bookieMappingCache = 
store.getMetadataCache(BookiesRackConfiguration.class);
+store.registerListener(this::handleUpdates);
+bookieMappingCache.get(BOOKIE_INFO_ROOT_PATH)

Review Comment:
   @rdhabalia 
   
   >infact this PR should not be merged.
   
   This is solving a problem your PR was not addressing, the use of blocking 
call in the notification watcher.
   
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Allow broker's service clusterIP customisation [pulsar-helm-chart]

2024-06-05 Thread via GitHub 


MMirelli commented on PR #498:
URL: 
https://github.com/apache/pulsar-helm-chart/pull/498#issuecomment-2150861369

   Anyway I did notice today, that using non-headless seems to have some 
advantages. With headless broker service, today I observed re-connections when 
using `pulsar-client`, that I didn't see in non-headless mode. It seems to be a 
configuration to play a bit with :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [fix] [client] Fix pattern consumer create crash if a part of partitions of a topic have been deleted [pulsar]

2024-06-05 Thread via GitHub 


lhotari commented on code in PR #22854:
URL: https://github.com/apache/pulsar/pull/22854#discussion_r1628364294


##
pulsar-common/src/main/java/org/apache/pulsar/common/naming/TopicName.java:
##
@@ -102,6 +102,10 @@ public static boolean isValid(String topic) {
 }
 }
 
+public static String getPartitionPattern(String topic) {
+return "^" + get(topic).getPartitionedTopicName().toString() + 
"-partition-[0-9]+$";

Review Comment:
   Use Pattern.quote to escape possible pattern characters in the topic name



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [fix] [client] Fix pattern consumer create crash if a part of partitions of a topic have been deleted [pulsar]

2024-06-05 Thread via GitHub 


lhotari commented on code in PR #22854:
URL: https://github.com/apache/pulsar/pull/22854#discussion_r1628361663


##
pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java:
##
@@ -1573,4 +1541,51 @@ protected void setCurrentReceiverQueueSize(int newSize) {
 CURRENT_RECEIVER_QUEUE_SIZE_UPDATER.set(this, newSize);
 resumeReceivingFromPausedConsumersIfNeeded();
 }
+
+/**
+ * Get the exists partitions of a partitioned topic, the result does not 
contain the partitions which has not been
+ * created yet(in other words, the partitions that do not exist in the 
response of "pulsar-admin topics list").
+ * @return sorted partitions list if it is a partitioned topic; @return an 
empty list if it is a non-partitioned
+ * topic.
+ */
+private CompletableFuture> getExistsPartitions(String topic) 
{
+TopicName topicName = TopicName.get(topic);
+if (!topicName.isPersistent()) {
+return FutureUtil.failedFuture(new IllegalArgumentException("The 
method getExistsPartitions"
++ " does not support non-persistent topic yet."));
+}
+return 
client.getLookup().getTopicsUnderNamespace(topicName.getNamespaceObject(),
+CommandGetTopicsOfNamespace.Mode.PERSISTENT,
+"^" + topicName.getPartitionedTopicName() + "$",

Review Comment:
   If this is intended to be an exact match, you might want to use 
Pattern.quote to escape any regex pattern characters in the topic name



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Allow broker's service clusterIP customisation [pulsar-helm-chart]

2024-06-05 Thread via GitHub 


MMirelli commented on PR #498:
URL: 
https://github.com/apache/pulsar-helm-chart/pull/498#issuecomment-2150845242

   Actually there doesn't seem to be a problem with broker service being 
headless. It must have been some other configuration that broke it.  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] [fix] [client] Fix pattern consumer create crash if a part of partitions of a topic have been deleted [pulsar]

2024-06-05 Thread via GitHub 


poorbarcode opened a new pull request, #22854:
URL: https://github.com/apache/pulsar/pull/22854

   ### Motivation
   
    Background
   
   **1.Regarding the API getting topics by regexp pattern, there are two 
implementations:***
   - `List HttpLookupService.getPartitionedTopicMetadata(...)`
   - `CommandGetTopicsOfNamespace 
BinaryProtoLookupService.getPartitionedTopicMetadata(...)`
   
   Pulsar transferred both response types `List` and 
`CommandGetTopicsOfNamespace` to a `GetTopicsResult` object. And discarded the 
partition information when doing transference. For example:
   - Get a list `topic-1-partition-0`, `topic-1-partition-1`.
   - The transferring operation will group them to `topic-1`.
   
   **2.The behavior of Patten consumers**
   - Before https://github.com/apache/pulsar/pull/5230, Patten consumers will 
try to create the missing partitions when it starts up.
   - After https://github.com/apache/pulsar/pull/5230, Patten consumers only 
subscribe to the partitions that exist to lead the topics can be deleted 
automatically. In other words, it sets the variable `createTopicIfDoesNotExist` 
of Multi Topics Consumer to `false`.
   
   ---
   
    Issue
   1. When users are starting a Pattern consumer, the consumer will try to 
subscribe to all the partitions even if some partitions have been deleted 
before, and then the client crashes due to a Topic Not Exists Exception. You 
can reproduce the issue by the test 
[testConsumerAfterOnePartDeleted](https://github.com/poorbarcode/pulsar/pull/91/files#diff-c6bb7cf6a6d6f5c4673ec5837351d981c835024fbbb864f32113c05b8ab30b4eR142)
   2. The Patten consumer that is started removes all partitions when one 
partition was deleted, even if there are still half of the partitions exist.
   
   ---
   
   ### Modifications
   
   - Multi Topics Consumer only subscribes to the existing partitions if the 
config `createTopicIfDoesNotExist` is `false`.
   - The Patten consumer that is started only removes the partitions that were 
deleted.
   
   ### Documentation
   
   
   
   - [ ] `doc` 
   - [ ] `doc-required` 
   - [x] `doc-not-needed` 
   - [ ] `doc-complete` 
   
   ### Matching PR in forked repository
   
   PR in forked repository: x
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] [Bug] Pulsar RocksDB metadata driver causes JVM to crash in unit tests [pulsar]

2024-06-05 Thread via GitHub 


lhotari closed issue #19250: [Bug] Pulsar RocksDB metadata driver causes JVM to 
crash in unit tests
URL: https://github.com/apache/pulsar/issues/19250


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(pulsar) branch master updated: [fix][meta] Check if metadata store is closed in RocksdbMetadataStore (#22852)

2024-06-05 Thread lhotari 
This is an automated email from the ASF dual-hosted git repository.

lhotari pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 74192871ed0 [fix][meta] Check if metadata store is closed in 
RocksdbMetadataStore (#22852)
74192871ed0 is described below

commit 74192871ed00870e5181a5bd4018ba196fd8f698
Author: Lari Hotari 
AuthorDate: Wed Jun 5 22:02:43 2024 +0300

[fix][meta] Check if metadata store is closed in RocksdbMetadataStore 
(#22852)
---
 .../metadata/impl/AbstractMetadataStore.java   | 25 +++---
 .../pulsar/metadata/impl/RocksdbMetadataStore.java | 15 +
 2 files changed, 27 insertions(+), 13 deletions(-)

diff --git 
a/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/AbstractMetadataStore.java
 
b/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/AbstractMetadataStore.java
index fa827bb40e7..7315e6a04a2 100644
--- 
a/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/AbstractMetadataStore.java
+++ 
b/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/AbstractMetadataStore.java
@@ -257,8 +257,7 @@ public abstract class AbstractMetadataStore implements 
MetadataStoreExtended, Co
 @Override
 public CompletableFuture> get(String path) {
 if (isClosed()) {
-return FutureUtil.failedFuture(
-new MetadataStoreException.AlreadyClosedException());
+return alreadyClosedFailedFuture();
 }
 long start = System.currentTimeMillis();
 if (!isValidPath(path)) {
@@ -286,8 +285,7 @@ public abstract class AbstractMetadataStore implements 
MetadataStoreExtended, Co
 @Override
 public final CompletableFuture> getChildren(String path) {
 if (isClosed()) {
-return FutureUtil.failedFuture(
-new MetadataStoreException.AlreadyClosedException());
+return alreadyClosedFailedFuture();
 }
 if (!isValidPath(path)) {
 return FutureUtil.failedFuture(new 
MetadataStoreException.InvalidPathException(path));
@@ -298,8 +296,7 @@ public abstract class AbstractMetadataStore implements 
MetadataStoreExtended, Co
 @Override
 public final CompletableFuture exists(String path) {
 if (isClosed()) {
-return FutureUtil.failedFuture(
-new MetadataStoreException.AlreadyClosedException());
+return alreadyClosedFailedFuture();
 }
 if (!isValidPath(path)) {
 return FutureUtil.failedFuture(new 
MetadataStoreException.InvalidPathException(path));
@@ -362,8 +359,7 @@ public abstract class AbstractMetadataStore implements 
MetadataStoreExtended, Co
 public final CompletableFuture delete(String path, Optional 
expectedVersion) {
 log.info("Deleting path: {} (v. {})", path, expectedVersion);
 if (isClosed()) {
-return FutureUtil.failedFuture(
-new MetadataStoreException.AlreadyClosedException());
+return alreadyClosedFailedFuture();
 }
 long start = System.currentTimeMillis();
 if (!isValidPath(path)) {
@@ -414,8 +410,7 @@ public abstract class AbstractMetadataStore implements 
MetadataStoreExtended, Co
 public CompletableFuture deleteRecursive(String path) {
 log.info("Deleting recursively path: {}", path);
 if (isClosed()) {
-return FutureUtil.failedFuture(
-new MetadataStoreException.AlreadyClosedException());
+return alreadyClosedFailedFuture();
 }
 return getChildren(path)
 .thenCompose(children -> FutureUtil.waitForAll(
@@ -435,8 +430,7 @@ public abstract class AbstractMetadataStore implements 
MetadataStoreExtended, Co
 public final CompletableFuture put(String path, byte[] data, 
Optional optExpectedVersion,
 EnumSet options) {
 if (isClosed()) {
-return FutureUtil.failedFuture(
-new MetadataStoreException.AlreadyClosedException());
+return alreadyClosedFailedFuture();
 }
 long start = System.currentTimeMillis();
 if (!isValidPath(path)) {
@@ -516,10 +510,15 @@ public abstract class AbstractMetadataStore implements 
MetadataStoreExtended, Co
 }
 }
 
-private boolean isClosed() {
+protected boolean isClosed() {
 return isClosed.get();
 }
 
+protected static  CompletableFuture alreadyClosedFailedFuture() {
+return FutureUtil.failedFuture(
+new MetadataStoreException.AlreadyClosedException());
+}
+
 @Override
 public void close() throws Exception {
 executor.shutdownNow();
diff --git 
a/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/RocksdbMetadataStore.java

Re: [PR] [fix][meta] Check if metadata store is closed in RocksdbMetadataStore [pulsar]

2024-06-05 Thread via GitHub 


lhotari merged PR #22852:
URL: https://github.com/apache/pulsar/pull/22852


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [fix] [broker] Fix broker deadlock in metadata-callback thread due to single thread processing [pulsar]

2024-06-05 Thread via GitHub 


rdhabalia commented on PR #22841:
URL: https://github.com/apache/pulsar/pull/22841#issuecomment-2150734770

   I don't see any issue with this PR but I am not going to waste my time as 
other PRs are opened and I don't think one wants to review it. so, please feel 
free to close.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [fix] [broker] Fix broker deadlock by making blocking bk-create async during ml-creation [pulsar]

2024-06-05 Thread via GitHub 


rdhabalia commented on PR #22842:
URL: https://github.com/apache/pulsar/pull/22842#issuecomment-2150730161

   also, please let us know if you want us to just create an issue with info 
so, we don't waste our time in creating PR. I don't think it's a correct way to 
review and merge something without any discussion. I feel it's not a good 
practice.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [fix] [broker] Fix broker deadlock by making blocking bk-create async during ml-creation [pulsar]

2024-06-05 Thread via GitHub 


rdhabalia commented on PR #22842:
URL: https://github.com/apache/pulsar/pull/22842#issuecomment-2150707821

   @merlimat @lhotari #22846 PR should have not be merged and as I explained in 
that PR, this PR actually solves the problem.
   I think what we did was wrong and I didn't agree with it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Excessive memory allocation in OTel broker metrics [pulsar]

2024-06-05 Thread via GitHub 


dragosvictor commented on issue #22817:
URL: https://github.com/apache/pulsar/issues/22817#issuecomment-2150687074

   @lhotari Working on the same improvement for the topic stats, can you please 
reopen this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Excessive memory allocation in OTel broker metrics [pulsar]

2024-06-05 Thread via GitHub 


lhotari commented on issue #22817:
URL: https://github.com/apache/pulsar/issues/22817#issuecomment-2150684479

   Fixed by #22837


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Excessive memory allocation in OTel broker metrics [pulsar]

2024-06-05 Thread via GitHub 


lhotari closed issue #22817: Excessive memory allocation in OTel broker metrics
URL: https://github.com/apache/pulsar/issues/22817


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(pulsar) branch master updated: [fix] [broker] Fix Broker was failing to load stats-internal with broken schema ledger (#22845)

2024-06-05 Thread rdhabalia 
This is an automated email from the ASF dual-hosted git repository.

rdhabalia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 326e9fa731a [fix] [broker] Fix Broker was failing to load 
stats-internal with broken schema ledger (#22845)
326e9fa731a is described below

commit 326e9fa731ae17304621ab915e36d52a9b28a7a0
Author: Rajan Dhabalia 
AuthorDate: Wed Jun 5 11:19:12 2024 -0700

[fix] [broker] Fix Broker was failing to load stats-internal with broken 
schema ledger (#22845)
---
 .../apache/pulsar/broker/service/persistent/PersistentTopic.java   | 7 +++
 1 file changed, 7 insertions(+)

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
index 18e69250c16..2165247b161 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
@@ -58,6 +58,8 @@ import java.util.stream.Collectors;
 import javax.annotation.Nonnull;
 import lombok.Getter;
 import lombok.Value;
+import org.apache.bookkeeper.client.BKException.BKNoSuchLedgerExistsException;
+import 
org.apache.bookkeeper.client.BKException.BKNoSuchLedgerExistsOnMetadataServerException;
 import org.apache.bookkeeper.client.api.LedgerMetadata;
 import org.apache.bookkeeper.mledger.AsyncCallbacks;
 import org.apache.bookkeeper.mledger.AsyncCallbacks.AddEntryCallback;
@@ -2829,6 +2831,11 @@ public class PersistentTopic extends AbstractTopic 
implements Topic, AddEntryCal
 }).exceptionally(e -> {
 log.error("[{}] Failed to get ledger 
metadata for the schema ledger {}",
 topic, ledgerId, e);
+if ((e.getCause() instanceof 
BKNoSuchLedgerExistsOnMetadataServerException)
+|| (e.getCause() instanceof 
BKNoSuchLedgerExistsException)) {
+completableFuture.complete(null);
+return null;
+}
 completableFuture.completeExceptionally(e);
 return null;
 });

Re: [I] [Bug] Broker is failing to load stats-internal with broken schema ledger [pulsar]

2024-06-05 Thread via GitHub 


rdhabalia closed issue #22844: [Bug] Broker is failing to load stats-internal 
with broken schema ledger
URL: https://github.com/apache/pulsar/issues/22844


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [fix] [broker] Fix Broker was failing to load stats-internal with broken schema ledger [pulsar]

2024-06-05 Thread via GitHub 


rdhabalia merged PR #22845:
URL: https://github.com/apache/pulsar/pull/22845


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [fix] Remove blocking calls from BookieRackAffinityMapping [pulsar]

2024-06-05 Thread via GitHub 


rdhabalia commented on code in PR #22846:
URL: https://github.com/apache/pulsar/pull/22846#discussion_r1628220942


##
pulsar-broker-common/src/main/java/org/apache/pulsar/bookie/rackawareness/BookieRackAffinityMapping.java:
##
@@ -116,25 +115,32 @@ public synchronized void setConf(Configuration conf) {
 super.setConf(conf);
 MetadataStore store;
 try {
-store = createMetadataStore(conf);
-bookieMappingCache = 
store.getMetadataCache(BookiesRackConfiguration.class);
-store.registerListener(this::handleUpdates);
-racksWithHost = bookieMappingCache.get(BOOKIE_INFO_ROOT_PATH).get()
-.orElseGet(BookiesRackConfiguration::new);
-for (Map bookieMapping : 
racksWithHost.values()) {
-for (String address : bookieMapping.keySet()) {
-bookieAddressListLastTime.add(BookieId.parse(address));
-}
-if (LOG.isDebugEnabled()) {
-LOG.debug("BookieRackAffinityMapping init, 
bookieAddressListLastTime {}",
-bookieAddressListLastTime);
-}
-}
-updateRacksWithHost(racksWithHost);
-watchAvailableBookies();
-} catch (InterruptedException | ExecutionException | MetadataException 
e) {
+store = getMetadataStore(conf);
+} catch (MetadataException e) {
 throw new RuntimeException(METADATA_STORE_INSTANCE + " failed to 
init BookieId list");
 }
+
+bookieMappingCache = 
store.getMetadataCache(BookiesRackConfiguration.class);
+store.registerListener(this::handleUpdates);
+bookieMappingCache.get(BOOKIE_INFO_ROOT_PATH)

Review Comment:
   > The BK client creation is moved to a background thread and it's ok, 
thought the thread waiting for the BK client to be created is still blocked.
   
   No, this solves the issue. deadlock was happening at 
`org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.asyncOpen` at 
`ledgers.computeIfAbsent`
   Moving bk-client creation async will not hold the lock at concurrentHashMap 
and that will also not cause metadata-thread waiting. 
   
   infact this PR should not be merged. Making change in bk-client and 
releasing bk version takes time and not all previous version can take that 
change. so, we need a fix which can solve the problem and #22842 should do it.
   so, if there is no issue with #22842 then can we merge it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] [Bug] Deadlock in broker service while initializing bkClient [pulsar]

2024-06-05 Thread via GitHub 


lhotari commented on issue #22699:
URL: https://github.com/apache/pulsar/issues/22699#issuecomment-2150670605

   This will be fixed by #22853


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] [improve] Refactored BK ClientFactory to return futures [pulsar]

2024-06-05 Thread via GitHub 


merlimat opened a new pull request, #22853:
URL: https://github.com/apache/pulsar/pull/22853

   ### Motivation
   
   BK client creation can be blocking when it's configured with rack-awareness 
policy. 
   This, combined with per-namespace BK client policies, can lead to deadlock 
situations as explained in #22840
   
   ### Modifications
   
1. Changed the BK client factory to return a future for BK client creation
2. Internally create the BK client in background 
   
   ### Documentation
   
   
   
   - [ ] `doc` 
   - [ ] `doc-required` 
   - [x] `doc-not-needed` 
   - [ ] `doc-complete` 
   
   ### Matching PR in forked repository
   
   PR in forked repository: 
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(pulsar) branch master updated: [fix] Remove blocking calls from BookieRackAffinityMapping (#22846)

2024-06-05 Thread mmerli 
This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new aece67e35ec [fix] Remove blocking calls from BookieRackAffinityMapping 
(#22846)
aece67e35ec is described below

commit aece67e35ecec4a9d90a951b78cfc89ca6395054
Author: Matteo Merli 
AuthorDate: Wed Jun 5 10:49:00 2024 -0700

[fix] Remove blocking calls from BookieRackAffinityMapping (#22846)
---
 .../rackawareness/BookieRackAffinityMapping.java   | 44 +-
 .../IsolatedBookieEnsemblePlacementPolicy.java |  2 +-
 2 files changed, 28 insertions(+), 18 deletions(-)

diff --git 
a/pulsar-broker-common/src/main/java/org/apache/pulsar/bookie/rackawareness/BookieRackAffinityMapping.java
 
b/pulsar-broker-common/src/main/java/org/apache/pulsar/bookie/rackawareness/BookieRackAffinityMapping.java
index 983822f2294..4a5ff746f40 100644
--- 
a/pulsar-broker-common/src/main/java/org/apache/pulsar/bookie/rackawareness/BookieRackAffinityMapping.java
+++ 
b/pulsar-broker-common/src/main/java/org/apache/pulsar/bookie/rackawareness/BookieRackAffinityMapping.java
@@ -70,7 +70,7 @@ public class BookieRackAffinityMapping extends 
AbstractDNSToSwitchMapping
 private BookiesRackConfiguration racksWithHost = new 
BookiesRackConfiguration();
 private Map bookieInfoMap = new HashMap<>();
 
-public static MetadataStore createMetadataStore(Configuration conf) throws 
MetadataException {
+static MetadataStore getMetadataStore(Configuration conf) throws 
MetadataException {
 MetadataStore store;
 Object storeProperty = conf.getProperty(METADATA_STORE_INSTANCE);
 if (storeProperty != null) {
@@ -116,12 +116,20 @@ public class BookieRackAffinityMapping extends 
AbstractDNSToSwitchMapping
 super.setConf(conf);
 MetadataStore store;
 try {
-store = createMetadataStore(conf);
-bookieMappingCache = 
store.getMetadataCache(BookiesRackConfiguration.class);
-store.registerListener(this::handleUpdates);
-racksWithHost = bookieMappingCache.get(BOOKIE_INFO_ROOT_PATH).get()
-.orElseGet(BookiesRackConfiguration::new);
-for (Map bookieMapping : 
racksWithHost.values()) {
+store = getMetadataStore(conf);
+} catch (MetadataException e) {
+throw new RuntimeException(METADATA_STORE_INSTANCE + " failed to 
init BookieId list");
+}
+
+bookieMappingCache = 
store.getMetadataCache(BookiesRackConfiguration.class);
+store.registerListener(this::handleUpdates);
+
+try {
+var racksWithHost = bookieMappingCache.get(BOOKIE_INFO_ROOT_PATH)
+.thenApply(optRes -> 
optRes.orElseGet(BookiesRackConfiguration::new))
+.get();
+
+for (var bookieMapping : racksWithHost.values()) {
 for (String address : bookieMapping.keySet()) {
 bookieAddressListLastTime.add(BookieId.parse(address));
 }
@@ -131,10 +139,12 @@ public class BookieRackAffinityMapping extends 
AbstractDNSToSwitchMapping
 }
 }
 updateRacksWithHost(racksWithHost);
-watchAvailableBookies();
-} catch (InterruptedException | ExecutionException | MetadataException 
e) {
-throw new RuntimeException(METADATA_STORE_INSTANCE + " failed to 
init BookieId list");
+} catch (ExecutionException | InterruptedException e) {
+LOG.error("Failed to update rack info. ", e);
+throw new RuntimeException(e);
 }
+
+watchAvailableBookies();
 }
 
 private void watchAvailableBookies() {
@@ -145,13 +155,13 @@ public class BookieRackAffinityMapping extends 
AbstractDNSToSwitchMapping
 field.setAccessible(true);
 RegistrationClient registrationClient = (RegistrationClient) 
field.get(bookieAddressResolver);
 registrationClient.watchWritableBookies(versioned -> {
-try {
-racksWithHost = 
bookieMappingCache.get(BOOKIE_INFO_ROOT_PATH).get()
-.orElseGet(BookiesRackConfiguration::new);
-updateRacksWithHost(racksWithHost);
-} catch (InterruptedException | ExecutionException e) {
-LOG.error("Failed to update rack info. ", e);
-}
+bookieMappingCache.get(BOOKIE_INFO_ROOT_PATH)
+.thenApply(optRes -> 
optRes.orElseGet(BookiesRackConfiguration::new))
+.thenAccept(this::updateRacksWithHost)
+.exceptionally(ex -> {
+LOG.error("Failed to update rack info. ", ex);
+

Re: [PR] [fix] Remove blocking calls from BookieRackAffinityMapping [pulsar]

2024-06-05 Thread via GitHub 


merlimat commented on code in PR #22846:
URL: https://github.com/apache/pulsar/pull/22846#discussion_r1628182108


##
pulsar-broker-common/src/main/java/org/apache/pulsar/bookie/rackawareness/BookieRackAffinityMapping.java:
##
@@ -116,25 +115,32 @@ public synchronized void setConf(Configuration conf) {
 super.setConf(conf);
 MetadataStore store;
 try {
-store = createMetadataStore(conf);
-bookieMappingCache = 
store.getMetadataCache(BookiesRackConfiguration.class);
-store.registerListener(this::handleUpdates);
-racksWithHost = bookieMappingCache.get(BOOKIE_INFO_ROOT_PATH).get()
-.orElseGet(BookiesRackConfiguration::new);
-for (Map bookieMapping : 
racksWithHost.values()) {
-for (String address : bookieMapping.keySet()) {
-bookieAddressListLastTime.add(BookieId.parse(address));
-}
-if (LOG.isDebugEnabled()) {
-LOG.debug("BookieRackAffinityMapping init, 
bookieAddressListLastTime {}",
-bookieAddressListLastTime);
-}
-}
-updateRacksWithHost(racksWithHost);
-watchAvailableBookies();
-} catch (InterruptedException | ExecutionException | MetadataException 
e) {
+store = getMetadataStore(conf);
+} catch (MetadataException e) {
 throw new RuntimeException(METADATA_STORE_INSTANCE + " failed to 
init BookieId list");
 }
+
+bookieMappingCache = 
store.getMetadataCache(BookiesRackConfiguration.class);
+store.registerListener(this::handleUpdates);
+bookieMappingCache.get(BOOKIE_INFO_ROOT_PATH)

Review Comment:
   merging this for now since it's solves a (small) part of the problem



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [fix] Remove blocking calls from BookieRackAffinityMapping [pulsar]

2024-06-05 Thread via GitHub 


merlimat merged PR #22846:
URL: https://github.com/apache/pulsar/pull/22846


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [fix] Remove blocking calls from BookieRackAffinityMapping [pulsar]

2024-06-05 Thread via GitHub 


merlimat commented on code in PR #22846:
URL: https://github.com/apache/pulsar/pull/22846#discussion_r1628179439


##
pulsar-broker-common/src/main/java/org/apache/pulsar/bookie/rackawareness/BookieRackAffinityMapping.java:
##
@@ -116,25 +115,32 @@ public synchronized void setConf(Configuration conf) {
 super.setConf(conf);
 MetadataStore store;
 try {
-store = createMetadataStore(conf);
-bookieMappingCache = 
store.getMetadataCache(BookiesRackConfiguration.class);
-store.registerListener(this::handleUpdates);
-racksWithHost = bookieMappingCache.get(BOOKIE_INFO_ROOT_PATH).get()
-.orElseGet(BookiesRackConfiguration::new);
-for (Map bookieMapping : 
racksWithHost.values()) {
-for (String address : bookieMapping.keySet()) {
-bookieAddressListLastTime.add(BookieId.parse(address));
-}
-if (LOG.isDebugEnabled()) {
-LOG.debug("BookieRackAffinityMapping init, 
bookieAddressListLastTime {}",
-bookieAddressListLastTime);
-}
-}
-updateRacksWithHost(racksWithHost);
-watchAvailableBookies();
-} catch (InterruptedException | ExecutionException | MetadataException 
e) {
+store = getMetadataStore(conf);
+} catch (MetadataException e) {
 throw new RuntimeException(METADATA_STORE_INSTANCE + " failed to 
init BookieId list");
 }
+
+bookieMappingCache = 
store.getMetadataCache(BookiesRackConfiguration.class);
+store.registerListener(this::handleUpdates);
+bookieMappingCache.get(BOOKIE_INFO_ROOT_PATH)

Review Comment:
   I think it does not solve the problem completely because the calling thread 
is still blocked. 
   
   The BK client creation is moved to a background thread and it's ok, thought 
the thread waiting for the BK client to be created is still blocked.
   
   eg: 
1. IO thread is initiating the creation of topic
2. We trigger creation of BK client
3. background-thread is creating BK client
4. IO thread is still blocked waiting for completion.
   
   I'm working on a change to return future from BK client factory that would 
avoid the problem.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [fix] [broker] Fix broker deadlock in metadata-callback thread due to single thread processing [pulsar]

2024-06-05 Thread via GitHub 


rdhabalia commented on PR #22841:
URL: https://github.com/apache/pulsar/pull/22841#issuecomment-2150608805

   I see. but `BookkeeperFactoryForCustomEnsemblePlacementPolicy` is anyway 
internal interface and if we feel any improvement then we can always change it 
without breaking anything. I think main root cause is BK-Client as everything 
on top of that follows the same sync pattern and we can make required 
enhancement after updating bk-client interface.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] [fix][metadata] Check if metadata store is closed in RocksdbMetadataStore [pulsar]

2024-06-05 Thread via GitHub 


lhotari opened a new pull request, #22852:
URL: https://github.com/apache/pulsar/pull/22852

   Fixes #19250
   
   ### Motivation
   
   Accessing a closed RocksDb database will crash the JVM. See #19250
   
   ### Modifications
   
   - Check if the metadata store is closed after acquiring the read lock.
   
   ### Documentation
   
   
   
   - [ ] `doc` 
   - [ ] `doc-required` 
   - [x] `doc-not-needed` 
   - [ ] `doc-complete` 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [fix] Remove blocking calls from BookieRackAffinityMapping [pulsar]

2024-06-05 Thread via GitHub 


rdhabalia commented on code in PR #22846:
URL: https://github.com/apache/pulsar/pull/22846#discussion_r1628166465


##
pulsar-broker-common/src/main/java/org/apache/pulsar/bookie/rackawareness/BookieRackAffinityMapping.java:
##
@@ -116,25 +115,32 @@ public synchronized void setConf(Configuration conf) {
 super.setConf(conf);
 MetadataStore store;
 try {
-store = createMetadataStore(conf);
-bookieMappingCache = 
store.getMetadataCache(BookiesRackConfiguration.class);
-store.registerListener(this::handleUpdates);
-racksWithHost = bookieMappingCache.get(BOOKIE_INFO_ROOT_PATH).get()
-.orElseGet(BookiesRackConfiguration::new);
-for (Map bookieMapping : 
racksWithHost.values()) {
-for (String address : bookieMapping.keySet()) {
-bookieAddressListLastTime.add(BookieId.parse(address));
-}
-if (LOG.isDebugEnabled()) {
-LOG.debug("BookieRackAffinityMapping init, 
bookieAddressListLastTime {}",
-bookieAddressListLastTime);
-}
-}
-updateRacksWithHost(racksWithHost);
-watchAvailableBookies();
-} catch (InterruptedException | ExecutionException | MetadataException 
e) {
+store = getMetadataStore(conf);
+} catch (MetadataException e) {
 throw new RuntimeException(METADATA_STORE_INSTANCE + " failed to 
init BookieId list");
 }
+
+bookieMappingCache = 
store.getMetadataCache(BookiesRackConfiguration.class);
+store.registerListener(this::handleUpdates);
+bookieMappingCache.get(BOOKIE_INFO_ROOT_PATH)

Review Comment:
   @merlimat can you please review #22842 for now as it makes bk-client 
creation async and avoids blocking call at 
`org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.asyncOpen` which 
will prevent the deadlock.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] [Bug] consumers stops receiving new messages due to invalid blockedConsumerOnUnackedMsgs state [pulsar]

2024-06-05 Thread via GitHub 


lhotari commented on issue #22657:
URL: https://github.com/apache/pulsar/issues/22657#issuecomment-2150533309

   @180254 I experimented with some changes in 
https://github.com/lhotari/pulsar/pull/192 , I added test cases based on your 
work. There are multiple inconsistencies in handling the unacked message counts 
and blocking/unblocking dispatchers. The main gap in the experiment is the 
handling for negative acknowledgements. 
   The changes I made fixed the test cases, but I didn't run other tests to 
verify that there aren't regressions caused by the change. Most likely there 
are because of invalid negative ack handling in the experiment.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [fix] Remove blocking calls from BookieRackAffinityMapping [pulsar]

2024-06-05 Thread via GitHub 


merlimat commented on code in PR #22846:
URL: https://github.com/apache/pulsar/pull/22846#discussion_r1628064192


##
pulsar-broker-common/src/main/java/org/apache/pulsar/bookie/rackawareness/BookieRackAffinityMapping.java:
##
@@ -116,25 +115,32 @@ public synchronized void setConf(Configuration conf) {
 super.setConf(conf);
 MetadataStore store;
 try {
-store = createMetadataStore(conf);
-bookieMappingCache = 
store.getMetadataCache(BookiesRackConfiguration.class);
-store.registerListener(this::handleUpdates);
-racksWithHost = bookieMappingCache.get(BOOKIE_INFO_ROOT_PATH).get()
-.orElseGet(BookiesRackConfiguration::new);
-for (Map bookieMapping : 
racksWithHost.values()) {
-for (String address : bookieMapping.keySet()) {
-bookieAddressListLastTime.add(BookieId.parse(address));
-}
-if (LOG.isDebugEnabled()) {
-LOG.debug("BookieRackAffinityMapping init, 
bookieAddressListLastTime {}",
-bookieAddressListLastTime);
-}
-}
-updateRacksWithHost(racksWithHost);
-watchAvailableBookies();
-} catch (InterruptedException | ExecutionException | MetadataException 
e) {
+store = getMetadataStore(conf);
+} catch (MetadataException e) {
 throw new RuntimeException(METADATA_STORE_INSTANCE + " failed to 
init BookieId list");
 }
+
+bookieMappingCache = 
store.getMetadataCache(BookiesRackConfiguration.class);
+store.registerListener(this::handleUpdates);
+bookieMappingCache.get(BOOKIE_INFO_ROOT_PATH)

Review Comment:
   Ok, makes sense. I think we should still change the sync call in the watcher 
event. It's unlikely to have a cache miss there, though it's technically 
possible. 
   
   Without changing BK client constructor to reflect the blocking operation 
that happens, I think we could change the BK provider class to do the creation 
in background.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [fix] [broker] Fix broker deadlock in metadata-callback thread due to single thread processing [pulsar]

2024-06-05 Thread via GitHub 


merlimat commented on PR #22841:
URL: https://github.com/apache/pulsar/pull/22841#issuecomment-2150448235

   >umm.. which interface? The issue is the interface in BookKeeper client 
which doesn't support async call and I don't recall any such interface in 
BookKeeper introduced by me.
   
   `BookkeeperFactoryForCustomEnsemblePlacementPolicy`
   
   Sure, the main issue is in BK client, though this issue is only present when 
using different BK clients based on namespace policy. 
   
   This is an example of problematic code, try/catch masking a blocking 
operation: 
   
https://github.com/apache/pulsar/blob/342d88dd193bb85c0af91c5193b1422808a9c821/pulsar-broker/src/main/java/org/apache/pulsar/broker/ManagedLedgerClientFactory.java#L100-L109
   
   That also breaks the strict bookie assignment requirement that you were 
mentioning, because it will fallback to default BK client if there is any 
failure to read the bookie mapping from metadata. What's worse, it will not 
self-correct, and it will continue to use the wrong BK client indefinitely.
   
   ```java
   // find or create bk-client in cache for a specific ensemblePlacementPolicy
   if (ensemblePlacementPolicyConfig != null && 
ensemblePlacementPolicyConfig.getPolicyClass() != null) {
   bkClient = 
bkEnsemblePolicyToBkClientMap.computeIfAbsent(ensemblePlacementPolicyConfig, 
(key) -> {
   try {
   return bookkeeperProvider.create(conf, 
metadataStore, eventLoopGroup,
   
Optional.ofNullable(ensemblePlacementPolicyConfig.getPolicyClass()),
   
ensemblePlacementPolicyConfig.getProperties(), statsLogger);
   } catch (Exception e) {
   log.error("Failed to initialize bk-client for policy 
{}, properties {}",
   
ensemblePlacementPolicyConfig.getPolicyClass(),
   
ensemblePlacementPolicyConfig.getProperties(), e);
   }
   return this.defaultBkClient;
   });
   }
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [feat] PIP-188 Support blue-green migration [pulsar-client-go]

2024-06-05 Thread via GitHub 


RobertIndie merged PR #1210:
URL: https://github.com/apache/pulsar-client-go/pull/1210


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(pulsar) branch branch-3.3 updated: [fix][broker] Fix ProducerBusy issue due to incorrect userCreatedProducerCount on non-persistent topic (#22685)

2024-06-05 Thread zhaocong 
This is an automated email from the ASF dual-hosted git repository.

zhaocong pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/branch-3.3 by this push:
 new abe6d79510c [fix][broker] Fix ProducerBusy issue due to incorrect 
userCreatedProducerCount on non-persistent topic (#22685)
abe6d79510c is described below

commit abe6d79510c9c743b60a4cfcf319ee27557763ed
Author: hr <64506104+hrz...@users.noreply.github.com>
AuthorDate: Thu May 9 21:49:27 2024 +0800

[fix][broker] Fix ProducerBusy issue due to incorrect 
userCreatedProducerCount on non-persistent topic (#22685)

Co-authored-by: ruihongzhou 
(cherry picked from commit 253e6506ea2c5ccc6afe1117e311cf24685ce4e9)
---
 .../service/nonpersistent/NonPersistentTopic.java  | 10 --
 .../nonpersistent/NonPersistentTopicTest.java  | 22 ++
 2 files changed, 22 insertions(+), 10 deletions(-)

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopic.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopic.java
index d19aeaa4b0f..86eab3d38b0 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopic.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopic.java
@@ -18,7 +18,6 @@
  */
 package org.apache.pulsar.broker.service.nonpersistent;
 
-import static com.google.common.base.Preconditions.checkArgument;
 import static 
org.apache.bookkeeper.mledger.impl.cache.RangeEntryCacheManagerImpl.create;
 import static 
org.apache.pulsar.common.policies.data.BacklogQuota.BacklogQuotaType;
 import static 
org.apache.pulsar.common.protocol.Commands.DEFAULT_CONSUMER_EPOCH;
@@ -58,7 +57,6 @@ import 
org.apache.pulsar.broker.service.BrokerServiceException.UnsupportedVersio
 import org.apache.pulsar.broker.service.Consumer;
 import org.apache.pulsar.broker.service.Dispatcher;
 import org.apache.pulsar.broker.service.GetStatsOptions;
-import org.apache.pulsar.broker.service.Producer;
 import org.apache.pulsar.broker.service.Replicator;
 import org.apache.pulsar.broker.service.StreamingStats;
 import org.apache.pulsar.broker.service.Subscription;
@@ -249,14 +247,6 @@ public class NonPersistentTopic extends AbstractTopic 
implements Topic, TopicPol
 return false;
 }
 
-@Override
-public void removeProducer(Producer producer) {
-checkArgument(producer.getTopic() == this);
-if (producers.remove(producer.getProducerName(), producer)) {
-handleProducerRemoved(producer);
-}
-}
-
 @Override
 public CompletableFuture 
checkIfTransactionBufferRecoverCompletely(boolean isTxnEnabled) {
 return  CompletableFuture.completedFuture(null);
diff --git 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopicTest.java
 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopicTest.java
index b33381126e5..e2aec70fb11 100644
--- 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopicTest.java
+++ 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopicTest.java
@@ -18,11 +18,13 @@
  */
 package org.apache.pulsar.broker.service.nonpersistent;
 
+import java.lang.reflect.Field;
 import java.util.Optional;
 import java.util.UUID;
 import java.util.concurrent.CompletableFuture;
 import java.util.concurrent.TimeUnit;
 import lombok.Cleanup;
+import org.apache.pulsar.broker.service.AbstractTopic;
 import org.apache.pulsar.broker.service.BrokerTestBase;
 import org.apache.pulsar.broker.service.SubscriptionOption;
 import org.apache.pulsar.client.admin.PulsarAdminException;
@@ -250,4 +252,24 @@ public class NonPersistentTopicTest extends BrokerTestBase 
{
 Awaitility.waitAtMost(10, TimeUnit.SECONDS).pollInterval(1, 
TimeUnit.SECONDS)
 .until(() -> subscriptionMap.get(keySharedSubName) == null);
 }
+
+
+@Test
+public void testRemoveProducerOnNonPersistentTopic() throws Exception {
+final String topicName = "non-persistent://prop/ns-abc/topic_" + 
UUID.randomUUID();
+
+Producer producer = pulsarClient.newProducer()
+.topic(topicName)
+.create();
+
+NonPersistentTopic topic = (NonPersistentTopic) 
pulsar.getBrokerService().getTopicReference(topicName).get();
+Field field = 
AbstractTopic.class.getDeclaredField("userCreatedProducerCount");
+field.setAccessible(true);
+int userCreatedProducerCount = (int) field.get(topic);
+assertEquals(userCreatedProducerCount, 1);
+
+producer.close();
+userCreatedProducerCount = (int) field.get(topic);
+assertEquals(userCreatedProducerCount, 0);
+}
 }

Re: [D] Where is 3.2.x pulsar-client-kafka? [pulsar]

2024-06-05 Thread via GitHub 


GitHub user lhotari edited a comment on the discussion: Where is 3.2.x 
pulsar-client-kafka?

It is managed in https://github.com/apache/pulsar-adapters and released 
separately. There haven't been releases since 2.11.0 . It doesn't follow the 
Pulsar releases, but it should continue to be compatible with recent Pulsar 
client versions. 

To use a newer Pulsar client version, it's recommended to use the Pulsar BOM. 
Please see docs at 
https://pulsar.apache.org/docs/3.2.x/client-libraries-java-setup/#pulsar-bom 
how to use Pulsar BOM. pulsar-client-kafka is a Kafka compatibility wrapper 
over the Pulsar Java client. You might also need [to use the unshaded 
version](https://pulsar.apache.org/docs/3.2.x/adaptors-kafka/#use-the-pulsar-kafka-compatibility-wrapper-together-with-existing-kafka-client)
 of `pulsar-client-kafka` which has the artifact id 
`pulsar-client-kafka-original` .





GitHub link: 
https://github.com/apache/pulsar/discussions/22849#discussioncomment-9677334


This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org

Re: [D] Where is 3.2.x pulsar-client-kafka? [pulsar]

2024-06-05 Thread via GitHub 


GitHub user lhotari added a comment to the discussion: Where is 3.2.x 
pulsar-client-kafka?

It is managed in https://github.com/apache/pulsar-adapters and released 
separately. There haven't been releases since 2.11.0 . It doesn't follow the 
Pulsar releases, but it should continue to be compatible with recent Pulsar 
client versions. 
To use a newer Pulsar client version, it's recommended to use the Pulsar BOM. 
Please see docs at 
https://pulsar.apache.org/docs/3.2.x/client-libraries-java-setup/#pulsar-bom 
how to use Pulsar BOM. pulsar-client-kafka is a Kafka compatibility wrapper 
over the Pulsar Java client.


GitHub link: 
https://github.com/apache/pulsar/discussions/22849#discussioncomment-9677334


This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org

(pulsar-site) branch main updated: Add more git tips (#824)

2024-06-05 Thread lhotari 
This is an automated email from the ASF dual-hosted git repository.

lhotari pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/pulsar-site.git


The following commit(s) were added to refs/heads/main by this push:
 new b022f93c96ec Add more git tips (#824)
b022f93c96ec is described below

commit b022f93c96ec2e8eb2e499ca8dff99a1fc774770
Author: Lari Hotari 
AuthorDate: Wed Jun 5 14:20:42 2024 +0300

Add more git tips (#824)
---
 contribute/release-process.md   |   2 +-
 contribute/setup-building.md|   2 +-
 contribute/{setup-mergetool.md => setup-git.md} | 114 +---
 docusaurus.config.js|  13 ++-
 preview.sh  |   7 ++
 sidebarsDevelopment.js  |   2 +-
 6 files changed, 124 insertions(+), 16 deletions(-)

diff --git a/contribute/release-process.md b/contribute/release-process.md
index 830cbf77aae7..0167fd8c608c 100644
--- a/contribute/release-process.md
+++ b/contribute/release-process.md
@@ -119,7 +119,7 @@ Note that you should also stop the workflow for previous 
Pulsar versions that ar
 
 ### Cherry-picking changes scheduled for the release
 
-Before proceeding, ensure that you have [set up a Git 
mergetool](setup-mergetool.md). This tool is essential for resolving merge 
conflicts that may arise during the cherry-picking process.
+Before proceeding, ensure that you have [set up a Git 
mergetool](setup-git.md#mergetool). This tool is essential for resolving merge 
conflicts that may arise during the cherry-picking process.
 
 Use a search such as `is:merged is:pr label:release/3.0.3 
-label:cherry-picked/branch-3.0` to search for merged PRs that are scheduled 
for the release, but haven't yet been cherry-picked.
 It is necessary to handle cherry-picks in the same order as they have been 
merged in the master branch. Otherwise there will be unnecessary merge 
conflicts to resolve.
diff --git a/contribute/setup-building.md b/contribute/setup-building.md
index 6fbcce640b9f..889ab6ee5f6c 100644
--- a/contribute/setup-building.md
+++ b/contribute/setup-building.md
@@ -7,7 +7,7 @@ title: Setup and building
 
 | Dependency | Description 

|
 
||-|
-| Git| The source code of Pulsar is hosted on GitHub as a git 
repository. To work with the git repository, please [install 
git](https://git-scm.com/downloads). We highly recommend that you also [set up 
a Git mergetool](setup-mergetool.md) for resolving merge conflicts. |
+| Git| The source code of Pulsar is hosted on GitHub as a git 
repository. To work with the git repository, please [install 
git](https://git-scm.com/downloads). We highly recommend that you also [set up 
a Git mergetool](setup-git.md#mergetool) for resolving merge conflicts. |
 | JDK| The source code of Pulsar is primarily written in Java. 
Therefore, you need a working Java Development Kit (JDK) to build it. Pulsar 
requires [JDK 17](https://adoptium.net/temurin/releases/?version=17) to build. |
 | Maven  | The source code of Pulsar is managed by [Apache 
Maven](https://maven.apache.org/) The required Maven version is 3.6.1+. 

|
 | Zip| The build process requires Zip as a utility tool.   

|
diff --git a/contribute/setup-mergetool.md b/contribute/setup-git.md
similarity index 60%
rename from contribute/setup-mergetool.md
rename to contribute/setup-git.md
index cf7b425f3a02..dd577fc20127 100644
--- a/contribute/setup-mergetool.md
+++ b/contribute/setup-git.md
@@ -1,16 +1,102 @@
 ---
-id: setup-mergetool
-title: Setting up Git mergetool
+id: setup-git
+title: Setting up Git for maintenance of Pulsar
 ---
 
-## Merge conflict resolution tooling
+## Git configuration
 
-For Apache Pulsar core developers, handling git merge conflict resolution is 
necessary. 
-To efficiently resolve merge conflicts, setting up tools that assist in 
visualizing these conflicts and resolving them is essential. 
+### Tooling
+
+ Install `gh` command line tool for GitHub
+
+Installing with `brew`, for other package managers, follow instructions at 
https://cli.github.com/. 
+
+```shell
+brew install gh
+```
+
+After installing authenticate to GitHub
+
+```shell
+gh auth login
+```
+
+ Optional: Install `tig` command line UI for git
+

(pulsar-helm-chart) branch master updated: Add defaultPulsarImageRepository configuration (#503)

2024-06-05 Thread lhotari 
This is an automated email from the ASF dual-hosted git repository.

lhotari pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar-helm-chart.git


The following commit(s) were added to refs/heads/master by this push:
 new 47c2ac4  Add defaultPulsarImageRepository configuration (#503)
47c2ac4 is described below

commit 47c2ac442af163181e8eb4a9b327fae4124f2d18
Author: Lari Hotari 
AuthorDate: Wed Jun 5 14:20:16 2024 +0300

Add defaultPulsarImageRepository configuration (#503)

- makes it easier to use a custom image
---
 charts/pulsar/templates/_helpers.tpl |  2 +-
 charts/pulsar/values.yaml| 24 +---
 2 files changed, 18 insertions(+), 8 deletions(-)

diff --git a/charts/pulsar/templates/_helpers.tpl 
b/charts/pulsar/templates/_helpers.tpl
index 9c24c77..c41e213 100644
--- a/charts/pulsar/templates/_helpers.tpl
+++ b/charts/pulsar/templates/_helpers.tpl
@@ -126,5 +126,5 @@ imagePullSecrets:
 Create full image name
 */}}
 {{- define "pulsar.imageFullName" -}}
-{{- printf "%s:%s" .image.repository (.image.tag | default 
.root.Values.defaultPulsarImageTag | default .root.Chart.AppVersion) -}}
+{{- printf "%s:%s" (.image.repository | default 
.root.Values.defaultPulsarImageRepository) (.image.tag | default 
.root.Values.defaultPulsarImageTag | default .root.Chart.AppVersion) -}}
 {{- end -}}
diff --git a/charts/pulsar/values.yaml b/charts/pulsar/values.yaml
index 2f0ef2f..21cb554 100755
--- a/charts/pulsar/values.yaml
+++ b/charts/pulsar/values.yaml
@@ -130,6 +130,9 @@ components:
   # pulsar manager
   pulsar_manager: false
 
+# default image repository for pulsar images
+defaultPulsarImageRepository: apachepulsar/pulsar-all
+
 # default image tag for pulsar images
 # uses chart's appVersion when unspecified
 defaultPulsarImageTag:
@@ -142,32 +145,38 @@ images:
   # imagePullSecrets:
   #   - secretName
   zookeeper:
-repository: apachepulsar/pulsar-all
+# uses defaultPulsarImageRepository when unspecified
+repository:
 # uses defaultPulsarImageTag when unspecified
 tag:
 pullPolicy: IfNotPresent
   bookie:
-repository: apachepulsar/pulsar-all
+# uses defaultPulsarImageRepository when unspecified
+repository:
 # uses defaultPulsarImageTag when unspecified
 tag:
 pullPolicy: IfNotPresent
   autorecovery:
-repository: apachepulsar/pulsar-all
+# uses defaultPulsarImageRepository when unspecified
+repository:
 # uses defaultPulsarImageTag when unspecified
 tag:
 pullPolicy: IfNotPresent
   broker:
-repository: apachepulsar/pulsar-all
+# uses defaultPulsarImageRepository when unspecified
+repository:
 # uses defaultPulsarImageTag when unspecified
 tag:
 pullPolicy: IfNotPresent
   proxy:
-repository: apachepulsar/pulsar-all
+# uses defaultPulsarImageRepository when unspecified
+repository:
 # uses defaultPulsarImageTag when unspecified
 tag:
 pullPolicy: IfNotPresent
   functions:
-repository: apachepulsar/pulsar-all
+# uses defaultPulsarImageRepository when unspecified
+repository:
 # uses defaultPulsarImageTag when unspecified
 tag:
   pulsar_manager:
@@ -713,7 +722,8 @@ pulsar_metadata:
   component: pulsar-init
   image:
 # the image used for running `pulsar-cluster-initialize` job
-repository: apachepulsar/pulsar-all
+# uses defaultPulsarImageRepository when unspecified
+repository:
 # uses defaultPulsarImageTag when unspecified
 tag:
 pullPolicy: IfNotPresent

Re: [PR] Add defaultPulsarImageRepository configuration [pulsar-helm-chart]

2024-06-05 Thread via GitHub 


lhotari merged PR #503:
URL: https://github.com/apache/pulsar-helm-chart/pull/503


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Rename setup-mergetool.md to setup-git.md and add more git tips [pulsar-site]

2024-06-05 Thread via GitHub 


lhotari merged PR #824:
URL: https://github.com/apache/pulsar-site/pull/824


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(pulsar-helm-chart) branch master updated: Upgrade kube-prometheus-stack to 59.x.x (#502)

2024-06-05 Thread lhotari 
This is an automated email from the ASF dual-hosted git repository.

lhotari pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar-helm-chart.git


The following commit(s) were added to refs/heads/master by this push:
 new aebf5fb  Upgrade kube-prometheus-stack to 59.x.x (#502)
aebf5fb is described below

commit aebf5fb0d53a6cfa1334bb05b694127571fa81fc
Author: Lari Hotari 
AuthorDate: Wed Jun 5 14:20:07 2024 +0300

Upgrade kube-prometheus-stack to 59.x.x (#502)
---
 README.md | 16 +++-
 charts/pulsar/Chart.yaml  |  4 ++--
 .../upgrade_prometheus_operator_crds.sh   | 19 +++
 3 files changed, 24 insertions(+), 15 deletions(-)

diff --git a/README.md b/README.md
index 5b0069a..c84e1d0 100644
--- a/README.md
+++ b/README.md
@@ -245,6 +245,20 @@ helm upgrade -f pulsar.yaml \
 
 For more detailed information, see our 
[Upgrading](http://pulsar.apache.org/docs/helm-upgrade/) guide.
 
+## Upgrading from Helm Chart version 3.0.0-3.4.x to 3.5.0 version and above
+
+The kube-prometheus-stack version has been upgraded to 59.x.x in Pulsar Helm 
Chart version 3.5.0 .
+Before running "helm upgrade", you should first upgrade the Prometheus 
Operator CRDs as [instructed
+in kube-prometheus-stack upgrade 
notes](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#from-58x-to-59x).
 
+
+There's a script to run the required commands:
+```shell
+./scripts/kube-prometheus-stack/upgrade_prometheus_operator_crds.sh 0.74.0
+```
+
+After, this you can proceed with `helm upgrade`.
+
+
 ## Upgrading from Helm Chart version 3.0.0-3.2.x to 3.3.0 version and above
 
 The kube-prometheus-stack version has been upgraded to 56.x.x in Pulsar Helm 
Chart version 3.3.0 .
@@ -253,7 +267,7 @@ in kube-prometheus-stack upgrade 
notes](https://github.com/prometheus-community/
 
 There's a script to run the required commands:
 ```shell
-./scripts/kube-prometheus-stack/upgrade_prometheus_operator_crds.sh
+./scripts/kube-prometheus-stack/upgrade_prometheus_operator_crds.sh 0.71.0
 ```
 
 After, this you can proceed with `helm upgrade`.
diff --git a/charts/pulsar/Chart.yaml b/charts/pulsar/Chart.yaml
index 3485b83..6615f22 100644
--- a/charts/pulsar/Chart.yaml
+++ b/charts/pulsar/Chart.yaml
@@ -21,7 +21,7 @@ apiVersion: v2
 appVersion: "3.0.5"
 description: Apache Pulsar Helm chart for Kubernetes
 name: pulsar
-version: 3.4.1
+version: 3.5.0
 kubeVersion: ">=1.21.0-0"
 home: https://pulsar.apache.org
 sources:
@@ -33,6 +33,6 @@ maintainers:
 email: d...@pulsar.apache.org
 dependencies:
   - name: kube-prometheus-stack
-version: 56.x.x
+version: 59.x.x
 repository: https://prometheus-community.github.io/helm-charts
 condition: kube-prometheus-stack.enabled
diff --git a/scripts/kube-prometheus-stack/upgrade_prometheus_operator_crds.sh 
b/scripts/kube-prometheus-stack/upgrade_prometheus_operator_crds.sh
index 15148de..d762a83 100755
--- a/scripts/kube-prometheus-stack/upgrade_prometheus_operator_crds.sh
+++ b/scripts/kube-prometheus-stack/upgrade_prometheus_operator_crds.sh
@@ -19,16 +19,11 @@
 #
 
 # This script is used to upgrade the Prometheus Operator CRDs before running 
"helm upgrade"
-# source: 
https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#from-55x-to-56x
+# source: 
https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#upgrading-an-existing-release-to-a-new-major-version
 # "Run these commands to update the CRDs before applying the upgrade."
-# "--force-conflicts" is required to upgrade the CRDs. Following instructions 
from https://github.com/prometheus-community/helm-charts/issues/2489
-kubectl apply --server-side --force-conflicts -f 
https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.71.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagerconfigs.yaml
-kubectl apply --server-side --force-conflicts -f 
https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.71.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml
-kubectl apply --server-side --force-conflicts -f 
https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.71.0/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml
-kubectl apply --server-side --force-conflicts -f 
https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.71.0/example/prometheus-operator-crd/monitoring.coreos.com_probes.yaml
-kubectl apply --server-side --force-conflicts -f 
https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.71.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheusagents.yaml
-kubectl apply --server-side --force-conflicts -f

Re: [I] Node does not exist [pulsar-helm-chart]

2024-06-05 Thread via GitHub 


lhotari closed issue #464: Node does not exist
URL: https://github.com/apache/pulsar-helm-chart/issues/464


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Upgrade kube-prometheus-stack to 59.x.x [pulsar-helm-chart]

2024-06-05 Thread via GitHub 


lhotari merged PR #502:
URL: https://github.com/apache/pulsar-helm-chart/pull/502


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Brokers fail starting on GKE Autopilot [pulsar-helm-chart]

2024-06-05 Thread via GitHub 


lhotari commented on issue #416:
URL: 
https://github.com/apache/pulsar-helm-chart/issues/416#issuecomment-2149523973

   I'm assuming that this was a duplicate of issue #473, fixed by #474.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Brokers fail starting on GKE Autopilot [pulsar-helm-chart]

2024-06-05 Thread via GitHub 


lhotari commented on issue #416:
URL: 
https://github.com/apache/pulsar-helm-chart/issues/416#issuecomment-2149516686

   @theoribeiro Do you have a chance to test with latest Apache Pulsar Helm 
Chart release 3.4.1 . Please reopen the ticket if the problem reproduces with 
3.4.1.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Brokers fail starting on GKE Autopilot [pulsar-helm-chart]

2024-06-05 Thread via GitHub 


lhotari closed issue #416: Brokers fail starting on GKE Autopilot
URL: https://github.com/apache/pulsar-helm-chart/issues/416


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Node does not exist [pulsar-helm-chart]

2024-06-05 Thread via GitHub 


lhotari commented on issue #464:
URL: 
https://github.com/apache/pulsar-helm-chart/issues/464#issuecomment-2149509517

   @nise-wg2 Using a custom image will be simplified with #503.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] Add defaultPulsarImageRepository configuration [pulsar-helm-chart]

2024-06-05 Thread via GitHub 


lhotari opened a new pull request, #503:
URL: https://github.com/apache/pulsar-helm-chart/pull/503

   Fixes #464
   
   ### Motivation
   
   Currently using a custom image requires updating multiple keys in the 
values.yaml file.
   It's easy to miss updating all locations where `apachepulsar/pulsar-all` is 
referenced.
   
   ### Modifications
   
   - add defaultPulsarImageRepository configuration
   
   ### Verifying this change
   
   - [ ] Make sure that the change passes the CI checks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Pulsar manager service fails saying "2024-05-07 15:02:17,476 INFO exited: pulsar-manager-backend (terminated by SIGKILL; not expected)" [pulsar-helm-chart]

2024-06-05 Thread via GitHub 


lhotari closed issue #496: Pulsar manager service fails saying "2024-05-07 
15:02:17,476 INFO exited: pulsar-manager-backend (terminated by SIGKILL; not 
expected)"
URL: https://github.com/apache/pulsar-helm-chart/issues/496


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Pulsar manager service fails saying "2024-05-07 15:02:17,476 INFO exited: pulsar-manager-backend (terminated by SIGKILL; not expected)" [pulsar-helm-chart]

2024-06-05 Thread via GitHub 


lhotari commented on issue #496:
URL: 
https://github.com/apache/pulsar-helm-chart/issues/496#issuecomment-2149489543

   This might be fixed in 3.4.1 version of the Pulsar Helm chart. Please reopen 
the issue if the problem persists.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] PKIX path building failed: unable to find valid certification path to request to requested target [pulsar-helm-chart]

2024-06-05 Thread via GitHub 


lhotari closed issue #492: PKIX path building failed: unable to find valid 
certification path to request to requested target
URL: https://github.com/apache/pulsar-helm-chart/issues/492


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] pulsar-broker failed to start with both Liveness probe and Readiness probes failed: HTTP probe failed with statuscode: 404 [pulsar-helm-chart]

2024-06-05 Thread via GitHub 


lhotari commented on issue #501:
URL: 
https://github.com/apache/pulsar-helm-chart/issues/501#issuecomment-2149485310

   Resolving since this isn't a pulsar-helm-chart issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] pulsar-broker failed to start with both Liveness probe and Readiness probes failed: HTTP probe failed with statuscode: 404 [pulsar-helm-chart]

2024-06-05 Thread via GitHub 


lhotari closed issue #501: pulsar-broker failed to start with both Liveness 
probe and Readiness probes failed: HTTP probe failed with statuscode: 404
URL: https://github.com/apache/pulsar-helm-chart/issues/501


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(pulsar-site) branch main updated: Solves apache/pulsar/issues/22848 (#909)

2024-06-05 Thread lhotari 
This is an automated email from the ASF dual-hosted git repository.

lhotari pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/pulsar-site.git


The following commit(s) were added to refs/heads/main by this push:
 new 6e0a2a4956fb Solves apache/pulsar/issues/22848 (#909)
6e0a2a4956fb is described below

commit 6e0a2a4956fb052a685f3b873459d3908e2dabc4
Author: Alvaro <102966649+alvarostr...@users.noreply.github.com>
AuthorDate: Wed Jun 5 12:45:27 2024 +0200

Solves apache/pulsar/issues/22848 (#909)

Add information to unload the topic when you execute an earliest
replication

As mentioned in motivation of

[PIP-356](https://github.com/apache/pulsar/blob/master/pip/pip-356.md#motivation)
---
 docs/administration-geo.md | 2 +-
 versioned_docs/version-3.0.x/administration-geo.md | 2 +-
 versioned_docs/version-3.1.x/administration-geo.md | 2 +-
 versioned_docs/version-3.2.x/administration-geo.md | 2 +-
 4 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/docs/administration-geo.md b/docs/administration-geo.md
index aa54e5d34ee4..575d92610c2c 100644
--- a/docs/administration-geo.md
+++ b/docs/administration-geo.md
@@ -275,6 +275,6 @@ Using geo-replication to migrate data between clusters is a 
special use case of
 :::note
 
 * The replication starts from step 4, which means existing messages in your 
old cluster are not replicated.
-* If you have some older messages to migrate, you can pre-create the 
replication subscriptions for each topic and set it at the earliest position by 
using `pulsar-admin topics create-subscription -s pulsar.repl.new-cluster -m 
earliest `.
+* If you have some older messages to migrate, you can pre-create the 
replication subscriptions for each topic and set it at the earliest position by 
using `pulsar-admin topics create-subscription -s pulsar.repl.new-cluster -m 
earliest `. Until 
[PIP-356](https://github.com/apache/pulsar/blob/master/pip/pip-356.md) is 
merged you will need to unload the topic to start georeplication.
 
 :::
diff --git a/versioned_docs/version-3.0.x/administration-geo.md 
b/versioned_docs/version-3.0.x/administration-geo.md
index 9be134cf196e..53043ba68b16 100644
--- a/versioned_docs/version-3.0.x/administration-geo.md
+++ b/versioned_docs/version-3.0.x/administration-geo.md
@@ -281,6 +281,6 @@ Using geo-replication to migrate data between clusters is a 
special use case of
 :::note
 
 * The replication starts from step 4, which means existing messages in your 
old cluster are not replicated.
-* If you have some older messages to migrate, you can pre-create the 
replication subscriptions for each topic and set it at the earliest position by 
using `pulsar-admin topics create-subscription -s pulsar.repl.new-cluster -m 
earliest `.
+* If you have some older messages to migrate, you can pre-create the 
replication subscriptions for each topic and set it at the earliest position by 
using `pulsar-admin topics create-subscription -s pulsar.repl.new-cluster -m 
earliest `. Until 
[PIP-356](https://github.com/apache/pulsar/blob/master/pip/pip-356.md) is 
merged you will need to unload the topic to start georeplication.
 
 :::
diff --git a/versioned_docs/version-3.1.x/administration-geo.md 
b/versioned_docs/version-3.1.x/administration-geo.md
index 8f3cd0aa31c8..798db7d494b9 100644
--- a/versioned_docs/version-3.1.x/administration-geo.md
+++ b/versioned_docs/version-3.1.x/administration-geo.md
@@ -274,6 +274,6 @@ Using geo-replication to migrate data between clusters is a 
special use case of
 :::note
 
 * The replication starts from step 4, which means existing messages in your 
old cluster are not replicated.
-* If you have some older messages to migrate, you can pre-create the 
replication subscriptions for each topic and set it at the earliest position by 
using `pulsar-admin topics create-subscription -s pulsar.repl.new-cluster -m 
earliest `.
+* If you have some older messages to migrate, you can pre-create the 
replication subscriptions for each topic and set it at the earliest position by 
using `pulsar-admin topics create-subscription -s pulsar.repl.new-cluster -m 
earliest `. Then you will need to unload the topic to start 
georeplication.
 
 :::
diff --git a/versioned_docs/version-3.2.x/administration-geo.md 
b/versioned_docs/version-3.2.x/administration-geo.md
index aa54e5d34ee4..575d92610c2c 100644
--- a/versioned_docs/version-3.2.x/administration-geo.md
+++ b/versioned_docs/version-3.2.x/administration-geo.md
@@ -275,6 +275,6 @@ Using geo-replication to migrate data between clusters is a 
special use case of
 :::note
 
 * The replication starts from step 4, which means existing messages in your 
old cluster are not replicated.
-* If you have some older messages to migrate, you can pre-create the 
replication subscriptions for each topic and set it at the earliest position by 
using `pulsar-admin topics create-subscription -s pulsar.repl.new-cluster -m

Re: [PR] Solves apache/pulsar/issues/22848 [pulsar-site]

2024-06-05 Thread via GitHub 


lhotari merged PR #909:
URL: https://github.com/apache/pulsar-site/pull/909


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] [Bug] pulsar-admin 2.x links don't properly link to 3.2 [pulsar]

2024-06-05 Thread via GitHub 


lhotari commented on issue #22828:
URL: https://github.com/apache/pulsar/issues/22828#issuecomment-2149442771

   A quick hack could be to add some reasonable page as the redirect target.
   Redirects could be made with Apache httpd syntax in 
https://github.com/apache/pulsar-site/blob/main/static/.htaccess


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] [Bug] consumers stops receiving new messages due to invalid blockedConsumerOnUnackedMsgs state [pulsar]

2024-06-05 Thread via GitHub 


dao-jun commented on issue #22657:
URL: https://github.com/apache/pulsar/issues/22657#issuecomment-2149440880

   I checked the code, my first vision is maybe it could have race conditions 
[here](https://github.com/apache/pulsar/blob/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java#L1021).
 But I don't dive deeper.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(pulsar) annotated tag v3.3.0 updated (c0aab493aaf -> 572794cb3d9)

2024-06-05 Thread zhaocong 
This is an automated email from the ASF dual-hosted git repository.

zhaocong pushed a change to annotated tag v3.3.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git


*** WARNING: tag v3.3.0 was modified! ***

from c0aab493aaf (commit)
  to 572794cb3d9 (tag)
 tagging c0aab493aafa5386dbf93c0b58a66a666aeba17a (commit)
 replaces v3.3.0-candidate-3
  by coderzc
  on Wed Jun 5 18:22:21 2024 +0800

- Log -
Release v3.3.0
-BEGIN PGP SIGNATURE-

iQJIBAABCAAyFiEE8s2tSUl/GoAOFzz+KNyu2EnEGA4FAmZgPF0UHHpoYW9jb25n
QGFwYWNoZS5vcmcACgkQKNyu2EnEGA4/Zw/+IqbQtI1Hr/Nau0YK+swDJEt1EBd4
+SXgPyjHRVgHhPImvxpjyLmR3Kif1n3MuKvIrgySVOP9nwGIeJfIs/cBSP1nZ6Gu
GVB9hbJRF0DEWrc73R1Nv9aHgEST3HsCMTEy+5o+di01FFCwtleO0ZMYbJUNCIIl
phMOr41S9TwjkQMA6tF1N1JgdRRdeZq8FHve1yUh0zVYKrjAyHbVOeJ+bHUR8ImA
YJCWCJzWg8UCAKfNbWRdp38kh9iTiEAW3KppKkappxJ9bpKBtzjyWMy8ktMr88Ab
kLnTjPgB0ZG78ScQHlC6qLLMV1RV/1qj9Alltz9UXpympBbCYMxV1UhbNPoCEmkH
ixIkOhsiDdr8AkAkQcqgyEe/n+rOBHYq9gYqHdnkorACeOEttFalDpCG7KY11bZX
qj6kYz7BCdFzWVhWQdTLY/WH+DrKGWEH95l+s3sIl+PogAh9bz4wo55Hgz550DSw
PXGEuFrKmqESyDo8RLB5Y7fHzEkJU846S+Wfw0L5uMpfcMNdOh9iZAJj8LX6iv2T
EpuM13XVZjBVnk21Zgc/B6Y0NAmm3POyEkYEJbE0cJ8Gl02lr5OQs4GzBcLxTOVS
h9K6c27nDEds1gIWk6v/Ihr3YUwuCQdJtE01D26UCyL8pTaModFwgVHf3j3l0zTt
IrDRMzK7+ZsWCP8=
=sX5P
-END PGP SIGNATURE-
---


No new revisions were added by this update.

Summary of changes:

Re: [I] [Bug] consumers stops receiving new messages due to invalid blockedConsumerOnUnackedMsgs state [pulsar]

2024-06-05 Thread via GitHub 


180254 commented on issue #22657:
URL: https://github.com/apache/pulsar/issues/22657#issuecomment-2149263694

   Before submitting a ticket, we also checked 3.2.x. The problem with our 
service was also recurring there.
   
   I reran the proposed BrokerServiceTest.java tests that I shared in previous 
messages.
   branch-3.0, commit 46b5419224c1a821ec0b9a839d0d7da57ac5f181: doesn't work, 
results as before
   master, commit 342d88dd193bb85c0af91c5193b1422808a9c821: doesn't work, 
results as before
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] [Bug] consumers stops receiving new messages due to invalid blockedConsumerOnUnackedMsgs state [pulsar]

2024-06-05 Thread via GitHub 


lhotari commented on issue #22657:
URL: https://github.com/apache/pulsar/issues/22657#issuecomment-2149156413

   > > pulsar version 3.2.1
   > 
   > @prasathsekar You might be facing another bug that is already fixed in 
3.2.3 with #22454. Please upgrade to Pulsar 3.2.3 and then comment whether the 
problem is resolved. 
   
   @MichalKoziorowski-TomTom please confirm whether this problem reproduces on 
3.2.3 . 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] [improve][cli] Use LoadManagerReport instead of Object [pulsar]

2024-06-05 Thread via GitHub 


crossoverJie opened a new pull request, #22850:
URL: https://github.com/apache/pulsar/pull/22850

   
   ### Modifications
   
   Use `LoadManagerReport` instead of Object.
   
   ### Verifying this change
   
   - [x] Make sure that the change passes the CI checks.
   
   *(Please pick either of the following options)*
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   *(or)*
   
   This change is already covered by existing tests, such as *(please describe 
tests)*.
   
   *(or)*
   
   This change added tests and can be verified as follows:
   
   *(example:)*
 - *Added integration tests for end-to-end deployment with large payloads 
(10MB)*
 - *Extended integration test for recovery after broker failure*
   
   ### Does this pull request potentially affect one of the following parts:
   
   
   
   *If the box was checked, please highlight the changes*
   
   - [ ] Dependencies (add or upgrade a dependency)
   - [ ] The public API
   - [ ] The schema
   - [ ] The default values of configurations
   - [ ] The threading model
   - [ ] The binary protocol
   - [ ] The REST endpoints
   - [ ] The admin CLI options
   - [ ] The metrics
   - [ ] Anything that affects deployment
   
   ### Documentation
   
   
   
   - [ ] `doc` 
   - [ ] `doc-required` 
   - [x] `doc-not-needed` 
   - [ ] `doc-complete` 
   
   ### Matching PR in forked repository
   
   PR in forked repository: https://github.com/crossoverJie/pulsar/pull/27


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

  1   2   >