Re: [PR] [Doc] Use absolute path when installing functions or using localrun [pulsar-site]
lhotari merged PR #877: URL: https://github.com/apache/pulsar-site/pull/877 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] receiver_queue_size is not honoured with partitioned topic [pulsar]
wallacepeng commented on issue #21593: URL: https://github.com/apache/pulsar/issues/21593#issuecomment-2033486574 for partition topic, the minimum receiver queue size is enforced to be at least the number of partitions . so your client will receive more messages . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [D] async/sync mode in pulsar producer and consumer [pulsar]
GitHub user ragaur-tibco added a comment to the discussion: async/sync mode in pulsar producer and consumer @lhotari I don't have any expectations or requirements I just want to know how to test the both modes any practical approach and anything in Pulsar SDK logs so that I can understand the internal process GitHub link: https://github.com/apache/pulsar/discussions/22382#discussioncomment-8991773 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
Re: [PR] [fix][cli] Fix help option [pulsar]
nodece commented on PR #22408: URL: https://github.com/apache/pulsar/pull/22408#issuecomment-2033473912 /pulsarbot rerun-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [fix][cli] Fix help option [pulsar]
nodece opened a new pull request, #22408: URL: https://github.com/apache/pulsar/pull/22408 ### Motivation Right now, the command with `-h` doesn't work: ``` ❯ ./bin/pulsar initialize-cluster-metadata -h picocli.CommandLine$MissingParameterException: Missing required options: '--cluster=', '--web-service-url=' Exception in thread "main" picocli.CommandLine$MissingParameterException: Missing required options: '--cluster=', '--web-service-url=' at picocli.CommandLine$MissingParameterException.create(CommandLine.java:18661) at picocli.CommandLine$MissingParameterException.access$21200(CommandLine.java:18649) at picocli.CommandLine$Interpreter.validateConstraints(CommandLine.java:13651) at picocli.CommandLine$Interpreter.parse(CommandLine.java:13614) at picocli.CommandLine$Interpreter.parse(CommandLine.java:13559) at picocli.CommandLine$Interpreter.parse(CommandLine.java:13454) at picocli.CommandLine.parseArgs(CommandLine.java:1552) at org.apache.pulsar.PulsarClusterMetadataSetup.main(PulsarClusterMetadataSetup.java:205) ``` Fixed version: ``` ❯ ./bin/pulsar initialize-cluster-metadata -h Usage: initialize-cluster-metadata [-gh] [-bn=] -c= [-cms=] [--existing-bk-metadata-service-uri=] [--initial-num-stream-storage-containers=] [--initial-num-transaction-coordinators=] [-md=] [-pp=] [-pu=] [-tb=] [-tw=] [-ub=] -uw= [--zookeeper-session-timeout-ms=] -bn, --default-namespace-bundle-number= The bundle numbers for the default namespaces (public/default), default is 16 Default: 0 -c, --cluster= Cluster name -cms, --configuration-metadata-store= Configuration Metadata Store connection string --existing-bk-metadata-service-uri= The metadata service URI of the existing BookKeeper cluster that you want to use -g, --generate-docs Generate docs -h, --helpShow this help message --initial-num-stream-storage-containers= Num storage containers of BookKeeper stream storage Default: 16 --initial-num-transaction-coordinators= Num transaction coordinators will assigned in cluster Default: 16 -md, --metadata-store= Metadata Store service url. eg: zk:my-zk:2181 -pp, --proxy-protocol= Proxy protocol to select type of routing at proxy. Possible Values: [SNI] -pu, --proxy-url= Proxy-server URL to which to connect. -tb, --broker-service-url-tls= Broker-service URL for new cluster with TLS encryption -tw, --web-service-url-tls= Web-service URL for new cluster with TLS encryption -ub, --broker-service-url= Broker-service URL for new cluster -uw, --web-service-url= Web-service URL for new cluster --zookeeper-session-timeout-ms= Local zookeeper session timeout ms Default: 3 ``` ### Modifications - Add `usageHelp=true` to the `-h, --help` option to print the help info. ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][client] client side memory leak when call ProducerBase.sendAsync(Message) [pulsar]
poorbarcode commented on code in PR #22393:
URL: https://github.com/apache/pulsar/pull/22393#discussion_r1548837735
##
pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java:
##
@@ -378,80 +378,95 @@ CompletableFuture internalSendAsync(Message
message) {
pendingMessagesUpDownCounter.increment();
pendingBytesUpDownCounter.add(msgSize);
-sendAsync(interceptorMessage, new SendCallback() {
-SendCallback nextCallback = null;
-MessageImpl nextMsg = null;
-long createdAt = System.nanoTime();
+sendAsync(interceptorMessage, new DefaultSendMessageCallback(future,
interceptorMessage, msgSize));
+return future;
+}
-@Override
-public CompletableFuture getFuture() {
-return future;
-}
+private class DefaultSendMessageCallback implements SendCallback {
-@Override
-public SendCallback getNextSendCallback() {
-return nextCallback;
-}
+CompletableFuture sendFuture;
+MessageImpl currentMsg;
+int msgSize;
+long createdAt = System.nanoTime();
+SendCallback nextCallback = null;
+MessageImpl nextMsg = null;
-@Override
-public MessageImpl getNextMessage() {
-return nextMsg;
-}
+DefaultSendMessageCallback(CompletableFuture sendFuture,
MessageImpl currentMsg, int msgSize) {
+this.sendFuture = sendFuture;
+this.currentMsg = currentMsg;
+this.msgSize = msgSize;
+}
-@Override
-public void sendComplete(Exception e) {
-long latencyNanos = System.nanoTime() - createdAt;
-pendingMessagesUpDownCounter.decrement();
-pendingBytesUpDownCounter.subtract(msgSize);
+@Override
+public CompletableFuture getFuture() {
+return sendFuture;
+}
+
+@Override
+public SendCallback getNextSendCallback() {
+return nextCallback;
+}
+
+@Override
+public MessageImpl getNextMessage() {
+return nextMsg;
+}
+@Override
+public void sendComplete(Exception e) {
+SendCallback loopingCallback = this;
+MessageImpl loopingMsg = currentMsg;
+while (loopingCallback != null) {
+onSendComplete(e, loopingCallback, loopingMsg);
+loopingMsg = loopingCallback.getNextMessage();
+loopingCallback = loopingCallback.getNextSendCallback();
+}
+}
+
+private void onSendComplete(Exception e, SendCallback sendCallback,
MessageImpl msg) {
+long createdAt = (sendCallback instanceof
ProducerImpl.DefaultSendMessageCallback)
+? ((DefaultSendMessageCallback) sendCallback).createdAt :
this.createdAt;
+long latencyNanos = System.nanoTime() - createdAt;
+pendingMessagesUpDownCounter.decrement();
+pendingBytesUpDownCounter.subtract(msgSize);
+ByteBuf payload = msg.getDataBuffer();
+if (e != null) {
+latencyHistogram.recordFailure(latencyNanos);
+stats.incrementSendFailed();
try {
-if (e != null) {
-latencyHistogram.recordFailure(latencyNanos);
-stats.incrementSendFailed();
-onSendAcknowledgement(interceptorMessage, null, e);
-future.completeExceptionally(e);
-} else {
-latencyHistogram.recordSuccess(latencyNanos);
-publishedBytesCounter.add(msgSize);
-onSendAcknowledgement(interceptorMessage,
interceptorMessage.getMessageId(), null);
-future.complete(interceptorMessage.getMessageId());
-stats.incrementNumAcksReceived(latencyNanos);
-}
+onSendAcknowledgement(msg, null, e);
+sendCallback.getFuture().completeExceptionally(e);
} finally {
-interceptorMessage.getDataBuffer().release();
+if (payload == null) {
+log.error("[{}] [{}] Payload is null when calling a
failed onSendComplete, which is not"
++ " expected.", topic, producerName);
+return;
+}
+ReferenceCountUtil.safeRelease(payload);
}
-
-while (nextCallback != null) {
-SendCallback sendCallback = nextCallback;
-MessageImpl msg = nextMsg;
-// Retain the buffer used by interceptors callback to get
message. Buffer will release after
-
Re: [PR] [fix][client] client side memory leak when call ProducerBase.sendAsync(Message) [pulsar]
gaoran10 commented on code in PR #22393:
URL: https://github.com/apache/pulsar/pull/22393#discussion_r1548837669
##
pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java:
##
@@ -418,16 +419,14 @@ public void sendComplete(Exception e) {
stats.incrementNumAcksReceived(latencyNanos);
}
} finally {
-interceptorMessage.getDataBuffer().release();
+ReferenceCountUtil.safeRelease(payloadInCurrentMsg);
Review Comment:
Yes.
Just a thought: if the publish latency contains the batch delay, the metric
can't accurately reflect the latency of communication with the broker, but
maybe it's OK because it also can reflect the trend.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
(pulsar) branch branch-3.2 updated: [improve][io]: Add validation for JDBC sink not supporting primitive schema (#22376)
This is an automated email from the ASF dual-hosted git repository.
technoboy pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 1639ae2a0ec [improve][io]: Add validation for JDBC sink not supporting
primitive schema (#22376)
1639ae2a0ec is described below
commit 1639ae2a0ec34ce475bb813f629a3ce97a3c5e14
Author: Baodi Shi
AuthorDate: Fri Mar 29 08:33:27 2024 +0800
[improve][io]: Add validation for JDBC sink not supporting primitive schema
(#22376)
---
pulsar-io/jdbc/core/pom.xml| 7 +
.../pulsar/io/jdbc/BaseJdbcAutoSchemaSink.java | 5
.../pulsar/io/jdbc/BaseJdbcAutoSchemaSinkTest.java | 25 +
.../apache/pulsar/io/jdbc/SqliteJdbcSinkTest.java | 31 +++---
4 files changed, 59 insertions(+), 9 deletions(-)
diff --git a/pulsar-io/jdbc/core/pom.xml b/pulsar-io/jdbc/core/pom.xml
index e1f15332a6c..7617e221105 100644
--- a/pulsar-io/jdbc/core/pom.xml
+++ b/pulsar-io/jdbc/core/pom.xml
@@ -71,6 +71,13 @@
provided
+
+ ${project.groupId}
+ pulsar-client-original
+ ${project.version}
+ test
+
+
\ No newline at end of file
diff --git
a/pulsar-io/jdbc/core/src/main/java/org/apache/pulsar/io/jdbc/BaseJdbcAutoSchemaSink.java
b/pulsar-io/jdbc/core/src/main/java/org/apache/pulsar/io/jdbc/BaseJdbcAutoSchemaSink.java
index 36c36740919..3655688c0f3 100644
---
a/pulsar-io/jdbc/core/src/main/java/org/apache/pulsar/io/jdbc/BaseJdbcAutoSchemaSink.java
+++
b/pulsar-io/jdbc/core/src/main/java/org/apache/pulsar/io/jdbc/BaseJdbcAutoSchemaSink.java
@@ -34,6 +34,7 @@ import org.apache.pulsar.client.api.schema.GenericObject;
import org.apache.pulsar.client.api.schema.GenericRecord;
import org.apache.pulsar.client.api.schema.KeyValueSchema;
import org.apache.pulsar.common.schema.KeyValue;
+import org.apache.pulsar.common.schema.SchemaType;
import org.apache.pulsar.functions.api.Record;
import org.apache.pulsar.io.jdbc.JdbcUtils.ColumnId;
@@ -137,6 +138,10 @@ public abstract class BaseJdbcAutoSchemaSink extends
JdbcAbstractSink data.get(k);
} else {
+SchemaType schemaType =
message.getSchema().getSchemaInfo().getType();
+if (schemaType.isPrimitive()) {
+throw new UnsupportedOperationException("Primitive schema is
not supported: " + schemaType);
+}
recordValueGetter = (key) -> ((GenericRecord)
record).getField(key);
}
String action = message.getProperties().get(ACTION_PROPERTY);
diff --git
a/pulsar-io/jdbc/core/src/test/java/org/apache/pulsar/io/jdbc/BaseJdbcAutoSchemaSinkTest.java
b/pulsar-io/jdbc/core/src/test/java/org/apache/pulsar/io/jdbc/BaseJdbcAutoSchemaSinkTest.java
index b15eb832242..c088dd3c42c 100644
---
a/pulsar-io/jdbc/core/src/test/java/org/apache/pulsar/io/jdbc/BaseJdbcAutoSchemaSinkTest.java
+++
b/pulsar-io/jdbc/core/src/test/java/org/apache/pulsar/io/jdbc/BaseJdbcAutoSchemaSinkTest.java
@@ -22,6 +22,10 @@ import java.util.function.Function;
import org.apache.avro.Schema;
import org.apache.avro.SchemaBuilder;
import org.apache.avro.util.Utf8;
+import org.apache.pulsar.client.api.schema.GenericObject;
+import org.apache.pulsar.client.api.schema.GenericRecord;
+import org.apache.pulsar.client.impl.schema.AutoConsumeSchema;
+import org.apache.pulsar.functions.api.Record;
import org.testng.Assert;
import org.testng.annotations.Test;
@@ -143,5 +147,26 @@ public class BaseJdbcAutoSchemaSinkTest {
return consumer.apply(record).endRecord().getFields().get(0).schema();
}
+@Test(expectedExceptions = UnsupportedOperationException.class,
+expectedExceptionsMessageRegExp = "Primitive schema is not
supported.*")
+@SuppressWarnings("unchecked")
+public void testNotSupportPrimitiveSchema() {
+BaseJdbcAutoSchemaSink baseJdbcAutoSchemaSink = new
BaseJdbcAutoSchemaSink() {};
+AutoConsumeSchema autoConsumeSchema = new AutoConsumeSchema();
+
autoConsumeSchema.setSchema(org.apache.pulsar.client.api.Schema.STRING);
+Record record = new Record() {
+@Override
+public org.apache.pulsar.client.api.Schema
getSchema() {
+return autoConsumeSchema;
+}
+
+@Override
+public GenericRecord getValue() {
+return null;
+}
+};
+baseJdbcAutoSchemaSink.createMutation((Record) record);
+}
+
}
\ No newline at end of file
diff --git
a/pulsar-io/jdbc/sqlite/src/test/java/org/apache/pulsar/io/jdbc/SqliteJdbcSinkTest.java
b/pulsar-io/jdbc/sqlite/src/test/java/org/apache/pulsar/io/jdbc/SqliteJdbcSinkTest.java
index d9ed4cbd442..ca01615bef1 100644
---
a/pulsar-io/jdbc/sqlite/src/test/java/org/apache/pulsar/io/jdbc/SqliteJdbcSinkTest.java
+++
(pulsar) branch branch-3.2 updated: [fix][ml] No rollover inactive ledgers when metadata service invalid (#22284)
This is an automated email from the ASF dual-hosted git repository.
technoboy pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new a83c892f1a9 [fix][ml] No rollover inactive ledgers when metadata
service invalid (#22284)
a83c892f1a9 is described below
commit a83c892f1a954564f6672253b8d533b020c0fce5
Author: houxiaoyu
AuthorDate: Sat Mar 30 21:38:55 2024 +0800
[fix][ml] No rollover inactive ledgers when metadata service invalid
(#22284)
### Motivation
We should not rollover inactive ledgers when metadata service is invailable.
### Modifications
Checking metadata service is vailable when schedule
`checkInactiveLedgerAndRollOver`
---
.../bookkeeper/mledger/impl/ManagedLedgerImpl.java | 7 ---
.../bookkeeper/mledger/impl/ManagedLedgerTest.java | 24 ++
2 files changed, 28 insertions(+), 3 deletions(-)
diff --git
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
index ee8e7c430ef..3d66bc8d6c0 100644
---
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
+++
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
@@ -4468,9 +4468,10 @@ public class ManagedLedgerImpl implements ManagedLedger,
CreateCallback {
@Override
public boolean checkInactiveLedgerAndRollOver() {
-long currentTimeMs = System.currentTimeMillis();
-if (currentLedgerEntries > 0 && inactiveLedgerRollOverTimeMs > 0 &&
currentTimeMs > (lastAddEntryTimeMs
-+ inactiveLedgerRollOverTimeMs)) {
+if (factory.isMetadataServiceAvailable()
+&& currentLedgerEntries > 0
+&& inactiveLedgerRollOverTimeMs > 0
+&& System.currentTimeMillis() > (lastAddEntryTimeMs +
inactiveLedgerRollOverTimeMs)) {
log.info("[{}] Closing inactive ledger, last-add entry {}", name,
lastAddEntryTimeMs);
if (STATE_UPDATER.compareAndSet(this, State.LedgerOpened,
State.ClosingLedger)) {
LedgerHandle currentLedger = this.currentLedger;
diff --git
a/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerTest.java
b/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerTest.java
index 0d9d6c0e573..9b4375dbedf 100644
---
a/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerTest.java
+++
b/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerTest.java
@@ -3922,6 +3922,30 @@ public class ManagedLedgerTest extends
MockedBookKeeperTestCase {
ledger.close();
}
+@Test
+public void testDontRollOverInactiveLedgersWhenMetadataServiceInvalid()
throws Exception {
+int inactiveLedgerRollOverTimeMs = 5;
+@Cleanup("shutdown")
+ManagedLedgerFactoryImpl factory = spy(new
ManagedLedgerFactoryImpl(metadataStore, bkc));
+// mock metadata service invalid
+when(factory.isMetadataServiceAvailable()).thenReturn(false);
+ManagedLedgerConfig config = new ManagedLedgerConfig();
+config.setInactiveLedgerRollOverTime(inactiveLedgerRollOverTimeMs,
TimeUnit.MILLISECONDS);
+ManagedLedgerImpl ledger = (ManagedLedgerImpl)
factory.open("rollover_inactive", config);
+
+long ledgerId = ledger.currentLedger.getId();
+
+Thread.sleep(inactiveLedgerRollOverTimeMs * 5);
+ledger.checkInactiveLedgerAndRollOver();
+
+Thread.sleep(inactiveLedgerRollOverTimeMs * 5);
+ledger.checkInactiveLedgerAndRollOver();
+
+assertEquals(ledger.currentLedger.getId(), ledgerId);
+
+ledger.close();
+}
+
@Test
public void testOffloadTaskCancelled() throws Exception {
@Cleanup("shutdown")
(pulsar) branch branch-3.2 updated: [improve] [broker] Servlet support response compression (#21667)
This is an automated email from the ASF dual-hosted git repository.
technoboy pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new c8591994807 [improve] [broker] Servlet support response compression
(#21667)
c8591994807 is described below
commit c859199480756eee5bf2dec1b3e6ebf2af9cec21
Author: Hang Chen
AuthorDate: Wed Mar 13 14:52:43 2024 +0800
[improve] [broker] Servlet support response compression (#21667)
---
.../org/apache/pulsar/broker/web/WebService.java | 16 +++--
.../apache/pulsar/broker/web/WebServiceTest.java | 72 ++
2 files changed, 82 insertions(+), 6 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebService.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebService.java
index 902593b7bf6..a7c42448990 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebService.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebService.java
@@ -44,6 +44,7 @@ import org.eclipse.jetty.server.handler.HandlerCollection;
import org.eclipse.jetty.server.handler.RequestLogHandler;
import org.eclipse.jetty.server.handler.ResourceHandler;
import org.eclipse.jetty.server.handler.StatisticsHandler;
+import org.eclipse.jetty.server.handler.gzip.GzipHandler;
import org.eclipse.jetty.servlet.FilterHolder;
import org.eclipse.jetty.servlet.ServletContextHandler;
import org.eclipse.jetty.servlet.ServletHolder;
@@ -258,15 +259,18 @@ public class WebService implements AutoCloseable {
public void addServlet(String path, ServletHolder servletHolder, boolean
requiresAuthentication,
Map attributeMap) {
-ServletContextHandler context = new
ServletContextHandler(ServletContextHandler.SESSIONS);
+ServletContextHandler servletContextHandler = new
ServletContextHandler(ServletContextHandler.SESSIONS);
// Notice: each context path should be unique, but there's nothing
here to verify that
-context.setContextPath(path);
-context.addServlet(servletHolder, MATCH_ALL);
+servletContextHandler.setContextPath(path);
+servletContextHandler.addServlet(servletHolder, MATCH_ALL);
if (attributeMap != null) {
-attributeMap.forEach(context::setAttribute);
+attributeMap.forEach(servletContextHandler::setAttribute);
}
-filterInitializer.addFilters(context, requiresAuthentication);
-handlers.add(context);
+filterInitializer.addFilters(servletContextHandler,
requiresAuthentication);
+
+GzipHandler gzipHandler = new GzipHandler();
+gzipHandler.setHandler(servletContextHandler);
+handlers.add(gzipHandler);
}
public void addStaticResources(String basePath, String resourcePath) {
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/web/WebServiceTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/web/WebServiceTest.java
index 405f3a11b5d..5386363373a 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/web/WebServiceTest.java
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/web/WebServiceTest.java
@@ -21,11 +21,14 @@ package org.apache.pulsar.broker.web;
import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertTrue;
import static org.testng.Assert.fail;
+
import com.google.common.collect.Multimap;
import com.google.common.collect.Sets;
import com.google.common.io.CharStreams;
import com.google.common.io.Closeables;
import io.netty.handler.ssl.util.InsecureTrustManagerFactory;
+
+import java.io.BufferedReader;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
@@ -42,6 +45,8 @@ import java.util.HashSet;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.ZipException;
import javax.net.ssl.HttpsURLConnection;
import javax.net.ssl.KeyManager;
import javax.net.ssl.KeyManagerFactory;
@@ -353,6 +358,73 @@ public class WebServiceTest {
assertEquals(res.getResponseBody(), "ok");
}
+@Test
+public void testCompressOutputMetricsInPrometheus() throws Exception {
+
+setupEnv(true, false, false, false, -1, false);
+
+String metricsUrl = pulsar.getWebServiceAddress() + "/metrics/";
+
+String[] command = {"curl", "-H", "Accept-Encoding: gzip", metricsUrl};
+
+ProcessBuilder processBuilder = new ProcessBuilder(command);
+Process process = processBuilder.start();
+
+InputStream inputStream = process.getInputStream();
+
+try {
+GZIPInputStream gzipInputStream = new GZIPInputStream(inputStream);
+
+// Process the decompressed content
+StringBuilder content = new StringBuilder();
+
(pulsar) branch branch-3.0 updated: [fix][broker] Skip topic.close during unloading if the topic future fails with ownership check, and fix isBundleOwnedByAnyBroker to use ns.checkOwnershipPresentAsyn
This is an automated email from the ASF dual-hosted git repository.
heesung pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 0404d6187ef [fix][broker] Skip topic.close during unloading if the
topic future fails with ownership check, and fix isBundleOwnedByAnyBroker to
use ns.checkOwnershipPresentAsync for ExtensibleLoadBalancer (#22379) (#22403)
0404d6187ef is described below
commit 0404d6187ef3c9db1ee2ca80e21058243b45e990
Author: Heesung Sohn <103456639+heesung...@users.noreply.github.com>
AuthorDate: Tue Apr 2 19:08:04 2024 -0700
[fix][broker] Skip topic.close during unloading if the topic future fails
with ownership check, and fix isBundleOwnedByAnyBroker to use
ns.checkOwnershipPresentAsync for ExtensibleLoadBalancer (#22379) (#22403)
---
.../extensions/manager/UnloadManager.java | 20 +-
.../pulsar/broker/service/BrokerService.java | 11 +-
.../pulsar/broker/web/PulsarWebResource.java | 7 +++-
.../ExtensibleLoadManagerImplBaseTest.java | 4 --
.../extensions/ExtensibleLoadManagerImplTest.java | 27 +
.../extensions/manager/UnloadManagerTest.java | 44 --
6 files changed, 86 insertions(+), 27 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/manager/UnloadManager.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/manager/UnloadManager.java
index ffdbbc2af42..bf9885b2a25 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/manager/UnloadManager.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/manager/UnloadManager.java
@@ -18,6 +18,8 @@
*/
package org.apache.pulsar.broker.loadbalance.extensions.manager;
+import static
org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigning;
+import static
org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
import static
org.apache.pulsar.broker.loadbalance.extensions.models.UnloadDecision.Label.Failure;
import static
org.apache.pulsar.broker.loadbalance.extensions.models.UnloadDecision.Reason.Unknown;
import java.util.Map;
@@ -25,6 +27,7 @@ import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.TimeUnit;
import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
import
org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState;
import
org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateData;
import org.apache.pulsar.broker.loadbalance.extensions.models.UnloadCounter;
@@ -88,14 +91,27 @@ public class UnloadManager implements StateChangeListener {
@Override
public void handleEvent(String serviceUnit, ServiceUnitStateData data,
Throwable t) {
-if (t != null && inFlightUnloadRequest.containsKey(serviceUnit)) {
+ServiceUnitState state = ServiceUnitStateData.state(data);
+
+if (StringUtils.isBlank(data.sourceBroker()) && (state == Owned ||
state == Assigning)) {
+if (log.isDebugEnabled()) {
+log.debug("Skipping {} for service unit {} from the assignment
command.", data, serviceUnit);
+}
+return;
+}
+
+if (t != null) {
if (log.isDebugEnabled()) {
log.debug("Handling {} for service unit {} with exception.",
data, serviceUnit, t);
}
this.complete(serviceUnit, t);
return;
}
-ServiceUnitState state = ServiceUnitStateData.state(data);
+
+if (log.isDebugEnabled()) {
+log.debug("Handling {} for service unit {}", data, serviceUnit);
+}
+
switch (state) {
case Free, Owned -> this.complete(serviceUnit, t);
default -> {
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
index fbec9bf413a..bfb289678d2 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
@@ -2277,9 +2277,18 @@ public class BrokerService implements Closeable {
}
closeFutures.add(topicFuture
.thenCompose(t -> t.isPresent() ?
t.get().close(closeWithoutWaitingClientDisconnect)
-: CompletableFuture.completedFuture(null)));
+: CompletableFuture.completedFuture(null))
+.exceptionally(e -> {
+if (e.getCause() instanceof
BrokerServiceException.ServiceUnitNotReadyException
+
Re: [PR] [fix][broker] Skip topic.close during unloading if the topic future fails with ownership check, and fix isBundleOwnedByAnyBroker to use ns.checkOwnershipPresentAsync for ExtensibleLoadBalance
heesung-sn merged PR #22403: URL: https://github.com/apache/pulsar/pull/22403 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][broker] Skip topic.close during unloading if the topic future fails with ownership check, and fix isBundleOwnedByAnyBroker to use ns.checkOwnershipPresentAsync for ExtensibleLoadBalance
heesung-sn merged PR #22406: URL: https://github.com/apache/pulsar/pull/22406 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch branch-3.1 updated: [fix][broker] Skip topic.close during unloading if the topic future fails with ownership check, and fix isBundleOwnedByAnyBroker to use ns.checkOwnershipPresentAsyn
This is an automated email from the ASF dual-hosted git repository.
heesung pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new e664432b7b7 [fix][broker] Skip topic.close during unloading if the
topic future fails with ownership check, and fix isBundleOwnedByAnyBroker to
use ns.checkOwnershipPresentAsync for ExtensibleLoadBalancer (#22379) (#22406)
e664432b7b7 is described below
commit e664432b7b7bf98e53afe6fe8fcb71e1aa664303
Author: Heesung Sohn <103456639+heesung...@users.noreply.github.com>
AuthorDate: Tue Apr 2 19:07:49 2024 -0700
[fix][broker] Skip topic.close during unloading if the topic future fails
with ownership check, and fix isBundleOwnedByAnyBroker to use
ns.checkOwnershipPresentAsync for ExtensibleLoadBalancer (#22379) (#22406)
---
.../extensions/manager/UnloadManager.java | 20 +-
.../pulsar/broker/service/BrokerService.java | 11 +-
.../pulsar/broker/web/PulsarWebResource.java | 5 +++
.../ExtensibleLoadManagerImplBaseTest.java | 4 --
.../extensions/ExtensibleLoadManagerImplTest.java | 27 +
.../extensions/manager/UnloadManagerTest.java | 44 --
6 files changed, 85 insertions(+), 26 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/manager/UnloadManager.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/manager/UnloadManager.java
index ffdbbc2af42..bf9885b2a25 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/manager/UnloadManager.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/manager/UnloadManager.java
@@ -18,6 +18,8 @@
*/
package org.apache.pulsar.broker.loadbalance.extensions.manager;
+import static
org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigning;
+import static
org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
import static
org.apache.pulsar.broker.loadbalance.extensions.models.UnloadDecision.Label.Failure;
import static
org.apache.pulsar.broker.loadbalance.extensions.models.UnloadDecision.Reason.Unknown;
import java.util.Map;
@@ -25,6 +27,7 @@ import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.TimeUnit;
import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
import
org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState;
import
org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateData;
import org.apache.pulsar.broker.loadbalance.extensions.models.UnloadCounter;
@@ -88,14 +91,27 @@ public class UnloadManager implements StateChangeListener {
@Override
public void handleEvent(String serviceUnit, ServiceUnitStateData data,
Throwable t) {
-if (t != null && inFlightUnloadRequest.containsKey(serviceUnit)) {
+ServiceUnitState state = ServiceUnitStateData.state(data);
+
+if (StringUtils.isBlank(data.sourceBroker()) && (state == Owned ||
state == Assigning)) {
+if (log.isDebugEnabled()) {
+log.debug("Skipping {} for service unit {} from the assignment
command.", data, serviceUnit);
+}
+return;
+}
+
+if (t != null) {
if (log.isDebugEnabled()) {
log.debug("Handling {} for service unit {} with exception.",
data, serviceUnit, t);
}
this.complete(serviceUnit, t);
return;
}
-ServiceUnitState state = ServiceUnitStateData.state(data);
+
+if (log.isDebugEnabled()) {
+log.debug("Handling {} for service unit {}", data, serviceUnit);
+}
+
switch (state) {
case Free, Owned -> this.complete(serviceUnit, t);
default -> {
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
index e08417a6498..9b11df6aeb9 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
@@ -2266,9 +2266,18 @@ public class BrokerService implements Closeable {
}
closeFutures.add(topicFuture
.thenCompose(t -> t.isPresent() ?
t.get().close(closeWithoutWaitingClientDisconnect)
-: CompletableFuture.completedFuture(null)));
+: CompletableFuture.completedFuture(null))
+.exceptionally(e -> {
+if (e.getCause() instanceof
BrokerServiceException.ServiceUnitNotReadyException
+
(pulsar) branch branch-3.2 updated: [fix][broker] Skip topic.close during unloading if the topic future fails with ownership check, and fix isBundleOwnedByAnyBroker to use ns.checkOwnershipPresentAsyn
This is an automated email from the ASF dual-hosted git repository.
heesung pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new d6ae0519d03 [fix][broker] Skip topic.close during unloading if the
topic future fails with ownership check, and fix isBundleOwnedByAnyBroker to
use ns.checkOwnershipPresentAsync for ExtensibleLoadBalancer (#22379) (#22407)
d6ae0519d03 is described below
commit d6ae0519d03a4442ec9fa2281b6a096a21f2dd9e
Author: Heesung Sohn <103456639+heesung...@users.noreply.github.com>
AuthorDate: Tue Apr 2 19:07:32 2024 -0700
[fix][broker] Skip topic.close during unloading if the topic future fails
with ownership check, and fix isBundleOwnedByAnyBroker to use
ns.checkOwnershipPresentAsync for ExtensibleLoadBalancer (#22379) (#22407)
---
.../extensions/manager/UnloadManager.java | 20 +-
.../pulsar/broker/service/BrokerService.java | 11 +-
.../pulsar/broker/web/PulsarWebResource.java | 5 +++
.../ExtensibleLoadManagerImplBaseTest.java | 4 --
.../extensions/ExtensibleLoadManagerImplTest.java | 27 +
.../extensions/manager/UnloadManagerTest.java | 44 --
6 files changed, 85 insertions(+), 26 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/manager/UnloadManager.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/manager/UnloadManager.java
index ffdbbc2af42..bf9885b2a25 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/manager/UnloadManager.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/manager/UnloadManager.java
@@ -18,6 +18,8 @@
*/
package org.apache.pulsar.broker.loadbalance.extensions.manager;
+import static
org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigning;
+import static
org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
import static
org.apache.pulsar.broker.loadbalance.extensions.models.UnloadDecision.Label.Failure;
import static
org.apache.pulsar.broker.loadbalance.extensions.models.UnloadDecision.Reason.Unknown;
import java.util.Map;
@@ -25,6 +27,7 @@ import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.TimeUnit;
import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
import
org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState;
import
org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateData;
import org.apache.pulsar.broker.loadbalance.extensions.models.UnloadCounter;
@@ -88,14 +91,27 @@ public class UnloadManager implements StateChangeListener {
@Override
public void handleEvent(String serviceUnit, ServiceUnitStateData data,
Throwable t) {
-if (t != null && inFlightUnloadRequest.containsKey(serviceUnit)) {
+ServiceUnitState state = ServiceUnitStateData.state(data);
+
+if (StringUtils.isBlank(data.sourceBroker()) && (state == Owned ||
state == Assigning)) {
+if (log.isDebugEnabled()) {
+log.debug("Skipping {} for service unit {} from the assignment
command.", data, serviceUnit);
+}
+return;
+}
+
+if (t != null) {
if (log.isDebugEnabled()) {
log.debug("Handling {} for service unit {} with exception.",
data, serviceUnit, t);
}
this.complete(serviceUnit, t);
return;
}
-ServiceUnitState state = ServiceUnitStateData.state(data);
+
+if (log.isDebugEnabled()) {
+log.debug("Handling {} for service unit {}", data, serviceUnit);
+}
+
switch (state) {
case Free, Owned -> this.complete(serviceUnit, t);
default -> {
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
index 6363e197616..a7130b4226e 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
@@ -2183,9 +2183,18 @@ public class BrokerService implements Closeable {
closeFutures.add(topicFuture
.thenCompose(t -> t.isPresent() ? t.get().close(
disconnectClients,
closeWithoutWaitingClientDisconnect)
-: CompletableFuture.completedFuture(null)));
+: CompletableFuture.completedFuture(null))
+.exceptionally(e -> {
+if (e.getCause() instanceof
Re: [PR] [fix][broker] Skip topic.close during unloading if the topic future fails with ownership check, and fix isBundleOwnedByAnyBroker to use ns.checkOwnershipPresentAsync for ExtensibleLoadBalance
heesung-sn merged PR #22407: URL: https://github.com/apache/pulsar/pull/22407 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch master updated: [fix][broker] Update TransferShedder underloaded broker check to consider max loaded broker's msgThroughputEMA and update IsExtensibleLoadBalancerImpl check (#22321)
This is an automated email from the ASF dual-hosted git repository.
heesung pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new d7d54522933 [fix][broker] Update TransferShedder underloaded broker
check to consider max loaded broker's msgThroughputEMA and update
IsExtensibleLoadBalancerImpl check (#22321)
d7d54522933 is described below
commit d7d54522933b63f6a74ec7139c6dedebe8ad9149
Author: Heesung Sohn <103456639+heesung...@users.noreply.github.com>
AuthorDate: Tue Apr 2 18:05:37 2024 -0700
[fix][broker] Update TransferShedder underloaded broker check to consider
max loaded broker's msgThroughputEMA and update IsExtensibleLoadBalancerImpl
check (#22321)
---
.../org/apache/pulsar/broker/PulsarService.java| 8 +++---
.../pulsar/broker/admin/impl/NamespacesBase.java | 4 +--
.../extensions/ExtensibleLoadManagerImpl.java | 6 +
.../extensions/scheduler/TransferShedder.java | 22 +++-
.../pulsar/broker/namespace/NamespaceService.java | 30 +++---
.../pulsar/broker/web/PulsarWebResource.java | 4 +--
.../extensions/scheduler/TransferShedderTest.java | 13 +-
7 files changed, 46 insertions(+), 41 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
index c1137bcfc25..9f7b40cc383 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
@@ -399,7 +399,7 @@ public class PulsarService implements AutoCloseable,
ShutdownService {
}
private void closeLeaderElectionService() throws Exception {
-if (ExtensibleLoadManagerImpl.isLoadManagerExtensionEnabled(config)) {
+if (ExtensibleLoadManagerImpl.isLoadManagerExtensionEnabled(this)) {
ExtensibleLoadManagerImpl.get(loadManager.get()).getLeaderElectionService().close();
} else {
if (this.leaderElectionService != null) {
@@ -1156,7 +1156,7 @@ public class PulsarService implements AutoCloseable,
ShutdownService {
}
protected void startLeaderElectionService() {
-if (ExtensibleLoadManagerImpl.isLoadManagerExtensionEnabled(config)) {
+if (ExtensibleLoadManagerImpl.isLoadManagerExtensionEnabled(this)) {
LOG.info("The load manager extension is enabled. Skipping
PulsarService LeaderElectionService.");
return;
}
@@ -1271,7 +1271,7 @@ public class PulsarService implements AutoCloseable,
ShutdownService {
LOG.info("Starting load management service ...");
this.loadManager.get().start();
-if (config.isLoadBalancerEnabled() &&
!ExtensibleLoadManagerImpl.isLoadManagerExtensionEnabled(config)) {
+if (config.isLoadBalancerEnabled() &&
!ExtensibleLoadManagerImpl.isLoadManagerExtensionEnabled(this)) {
LOG.info("Starting load balancer");
if (this.loadReportTask == null) {
long loadReportMinInterval =
config.getLoadBalancerReportUpdateMinIntervalMillis();
@@ -1358,7 +1358,7 @@ public class PulsarService implements AutoCloseable,
ShutdownService {
* @return a reference of the current LeaderElectionService
instance.
*/
public LeaderElectionService getLeaderElectionService() {
-if (ExtensibleLoadManagerImpl.isLoadManagerExtensionEnabled(config)) {
+if (ExtensibleLoadManagerImpl.isLoadManagerExtensionEnabled(this)) {
return
ExtensibleLoadManagerImpl.get(loadManager.get()).getLeaderElectionService();
} else {
return this.leaderElectionService;
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java
index 9d01530c601..bbadc7bb331 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java
@@ -969,13 +969,13 @@ public abstract class NamespacesBase extends
AdminResource {
return CompletableFuture.completedFuture(null);
})
.thenCompose(__ -> {
-if
(ExtensibleLoadManagerImpl.isLoadManagerExtensionEnabled(config())) {
+if
(ExtensibleLoadManagerImpl.isLoadManagerExtensionEnabled(pulsar())) {
return CompletableFuture.completedFuture(null);
}
return validateLeaderBrokerAsync();
})
.thenAccept(__ -> {
-if
(ExtensibleLoadManagerImpl.isLoadManagerExtensionEnabled(config())) {
+if
Re: [PR] [fix][broker] Update TransferShedder underloaded broker check to consider max loaded broker's msgThroughputEMA and update IsExtensibleLoadBalancerImpl check [pulsar]
heesung-sn merged PR #22321: URL: https://github.com/apache/pulsar/pull/22321 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][client] client side memory leak when call ProducerBase.sendAsync(Message) [pulsar]
codelipenghui commented on code in PR #22393:
URL: https://github.com/apache/pulsar/pull/22393#discussion_r1548779936
##
pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java:
##
@@ -378,80 +378,95 @@ CompletableFuture internalSendAsync(Message
message) {
pendingMessagesUpDownCounter.increment();
pendingBytesUpDownCounter.add(msgSize);
-sendAsync(interceptorMessage, new SendCallback() {
-SendCallback nextCallback = null;
-MessageImpl nextMsg = null;
-long createdAt = System.nanoTime();
+sendAsync(interceptorMessage, new DefaultSendMessageCallback(future,
interceptorMessage, msgSize));
+return future;
+}
-@Override
-public CompletableFuture getFuture() {
-return future;
-}
+private class DefaultSendMessageCallback implements SendCallback {
-@Override
-public SendCallback getNextSendCallback() {
-return nextCallback;
-}
+CompletableFuture sendFuture;
+MessageImpl currentMsg;
+int msgSize;
+long createdAt = System.nanoTime();
+SendCallback nextCallback = null;
+MessageImpl nextMsg = null;
-@Override
-public MessageImpl getNextMessage() {
-return nextMsg;
-}
+DefaultSendMessageCallback(CompletableFuture sendFuture,
MessageImpl currentMsg, int msgSize) {
+this.sendFuture = sendFuture;
+this.currentMsg = currentMsg;
+this.msgSize = msgSize;
+}
-@Override
-public void sendComplete(Exception e) {
-long latencyNanos = System.nanoTime() - createdAt;
-pendingMessagesUpDownCounter.decrement();
-pendingBytesUpDownCounter.subtract(msgSize);
+@Override
+public CompletableFuture getFuture() {
+return sendFuture;
+}
+
+@Override
+public SendCallback getNextSendCallback() {
+return nextCallback;
+}
+
+@Override
+public MessageImpl getNextMessage() {
+return nextMsg;
+}
+@Override
+public void sendComplete(Exception e) {
+SendCallback loopingCallback = this;
+MessageImpl loopingMsg = currentMsg;
+while (loopingCallback != null) {
+onSendComplete(e, loopingCallback, loopingMsg);
+loopingMsg = loopingCallback.getNextMessage();
+loopingCallback = loopingCallback.getNextSendCallback();
+}
+}
+
+private void onSendComplete(Exception e, SendCallback sendCallback,
MessageImpl msg) {
+long createdAt = (sendCallback instanceof
ProducerImpl.DefaultSendMessageCallback)
+? ((DefaultSendMessageCallback) sendCallback).createdAt :
this.createdAt;
+long latencyNanos = System.nanoTime() - createdAt;
+pendingMessagesUpDownCounter.decrement();
+pendingBytesUpDownCounter.subtract(msgSize);
+ByteBuf payload = msg.getDataBuffer();
+if (e != null) {
+latencyHistogram.recordFailure(latencyNanos);
+stats.incrementSendFailed();
try {
-if (e != null) {
-latencyHistogram.recordFailure(latencyNanos);
-stats.incrementSendFailed();
-onSendAcknowledgement(interceptorMessage, null, e);
-future.completeExceptionally(e);
-} else {
-latencyHistogram.recordSuccess(latencyNanos);
-publishedBytesCounter.add(msgSize);
-onSendAcknowledgement(interceptorMessage,
interceptorMessage.getMessageId(), null);
-future.complete(interceptorMessage.getMessageId());
-stats.incrementNumAcksReceived(latencyNanos);
-}
+onSendAcknowledgement(msg, null, e);
+sendCallback.getFuture().completeExceptionally(e);
} finally {
-interceptorMessage.getDataBuffer().release();
+if (payload == null) {
+log.error("[{}] [{}] Payload is null when calling a
failed onSendComplete, which is not"
++ " expected.", topic, producerName);
+return;
+}
+ReferenceCountUtil.safeRelease(payload);
}
-
-while (nextCallback != null) {
-SendCallback sendCallback = nextCallback;
-MessageImpl msg = nextMsg;
-// Retain the buffer used by interceptors callback to get
message. Buffer will release after
-
Re: [PR] [Doc] Use absolute path when installing functions or using localrun [pulsar-site]
lhotari commented on code in PR #877: URL: https://github.com/apache/pulsar-site/pull/877#discussion_r1548684505 ## docs/admin-api-functions.md: ## @@ -46,7 +46,7 @@ pulsar-admin functions create \ --inputs test-input-topic \ --output persistent://public/default/test-output-topic \ --classname org.apache.pulsar.functions.api.examples.ExclamationFunction \ ---jar /examples/api-examples.jar +--jar $PWD/examples/api-examples.jar Review Comment: The `/path/to/file.jar` type of examples have the problem that for the user, it's really a lot of extra steps to find out the absolute path for the file arguments. That's why I prefer `$PWD/file.jar` since it's simple and if someone doesn't already know it, they have the chance to learn it. `$PWD` is for a unix shell, but the examples are already assuming a unix shell since the syntax is all unix like. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [Doc] Use absolute path when installing functions or using localrun [pulsar-site]
lhotari commented on code in PR #877: URL: https://github.com/apache/pulsar-site/pull/877#discussion_r1548681330 ## docs/io-cli.md: ## @@ -81,8 +81,8 @@ pulsar-admin sources update options |Flag|Description| ||---| -| `-a`, `--archive` | The path to the NAR archive for the source. It also supports url-path (http/https/file [file protocol assumes that file already exists on worker host]) from which worker can download the package. -| `--classname` | The source's class name if `archive` is file-url-path (file://). +| `-a`, `--archive` | The absolute path to the NAR archive for the source. It also supports url-path (http/https/file [file protocol assumes that file already exists on worker host]) from which worker can download the package. +| `--classname` | The source's class name. Mandatory if `archive` doesn't reference a NAR file. Review Comment: Done. ## docs/io-cli.md: ## @@ -326,9 +326,9 @@ pulsar-admin sinks create options |Flag|Description| ||---| -| `-a`, `--archive` | The path to the archive file for the sink. It also supports url-path (http/https/file [file protocol assumes that file already exists on worker host]) from which worker can download the package. +| `-a`, `--archive` | The absolute path to the archive file for the sink. It also supports url-path (http/https/file [file protocol assumes that file already exists on worker host]) from which worker can download the package. | `--auto-ack` | Whether or not the framework will automatically acknowledge messages. -| `--classname` | The sink's class name if `archive` is file-url-path (file://). +| `--classname` | The sink's class name. Mandatory if `archive` doesn't reference a NAR file. Review Comment: Done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [Doc] Use absolute path when installing functions or using localrun [pulsar-site]
lhotari commented on code in PR #877: URL: https://github.com/apache/pulsar-site/pull/877#discussion_r1548681012 ## docs/io-cli.md: ## @@ -49,8 +49,8 @@ pulsar-admin sources create options |Flag|Description| ||---| -| `-a`, `--archive` | The path to the NAR archive for the source. It also supports url-path (http/https/file [file protocol assumes that file already exists on worker host]) from which worker can download the package. -| `--classname` | The source's class name if `archive` is file-url-path (file://). +| `-a`, `--archive` | The absolute path to the NAR archive for the source. It also supports url-path (http/https/file [file protocol assumes that file already exists on worker host]) from which worker can download the package. +| `--classname` | The source's class name. Mandatory if `archive` doesn't reference a NAR file. Review Comment: Done. ## docs/io-cli.md: ## @@ -245,9 +245,9 @@ pulsar-admin sources localrun options |Flag|Description| ||---| -| `-a`, `--archive` | The path to the NAR archive for the Source. It also supports url-path (http/https/file [file protocol assumes that file already exists on worker host]) from which worker can download the package. +| `-a`, `--archive` | The absolute path to the NAR archive for the Source. It also supports url-path (http/https/file [file protocol assumes that file already exists on worker host]) from which worker can download the package. | `--broker-service-url` | The URL for the Pulsar broker. -|`--classname`|The source's class name if `archive` is file-url-path (file://). +|`--classname`|The source's class name. Mandatory if `archive` doesn't reference a NAR file. Review Comment: Done. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [Doc] Use absolute path when installing functions or using localrun [pulsar-site]
lhotari commented on code in PR #877: URL: https://github.com/apache/pulsar-site/pull/877#discussion_r1548677950 ## docs/io-cli.md: ## @@ -49,8 +49,8 @@ pulsar-admin sources create options |Flag|Description| ||---| -| `-a`, `--archive` | The path to the NAR archive for the source. It also supports url-path (http/https/file [file protocol assumes that file already exists on worker host]) from which worker can download the package. -| `--classname` | The source's class name if `archive` is file-url-path (file://). +| `-a`, `--archive` | The absolute path to the NAR archive for the source. It also supports url-path (http/https/file [file protocol assumes that file already exists on worker host]) from which worker can download the package. +| `--classname` | The source's class name. Mandatory if `archive` doesn't reference a NAR file. Review Comment: > For now, I would recommend leaving it as "Mandatory" without any exclusions. Ok, I'll just make it "The source's class name". Without any other additions? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [Doc] Use absolute path when installing functions or using localrun [pulsar-site]
lhotari commented on code in PR #877: URL: https://github.com/apache/pulsar-site/pull/877#discussion_r1548676145 ## docs/admin-api-functions.md: ## @@ -46,7 +46,7 @@ pulsar-admin functions create \ --inputs test-input-topic \ --output persistent://public/default/test-output-topic \ --classname org.apache.pulsar.functions.api.examples.ExclamationFunction \ ---jar /examples/api-examples.jar +--jar $PWD/examples/api-examples.jar Review Comment: I think `$PWD` gives a good hint that it's an absolute path. At least I have found it useful with `pulsar-admin` when I have the files in the current directory and I want to reference the files with an absolute path. I guess there are also other ways. Perhaps it could be explained to the user that `$PWD` is the current working directory. They will also find out if they try it or google it. :) In this particular case, the original `/examples/api-examples.jar` doesn't seem to make much sense. That's why I added the `$PWD` prefix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][broker] Skip topic.close during unloading if the topic future fails with ownership check, and fix isBundleOwnedByAnyBroker to use ns.checkOwnershipPresentAsync for ExtensibleLoadBalance
github-actions[bot] commented on PR #22407: URL: https://github.com/apache/pulsar/pull/22407#issuecomment-2033175264 @heesung-sn Please add the following content to your PR description and select a checkbox: ``` - [ ] `doc` - [ ] `doc-required` - [ ] `doc-not-needed` - [ ] `doc-complete` ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [fix][broker] Skip topic.close during unloading if the topic future fails with ownership check, and fix isBundleOwnedByAnyBroker to use ns.checkOwnershipPresentAsync for ExtensibleLoadBalancer (#
heesung-sn opened a new pull request, #22407: URL: https://github.com/apache/pulsar/pull/22407 (cherry picked from commit 3eb3b1cd23d2cc11424bf882e244d3bc2e92bf27) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [Doc] Use absolute path when installing functions or using localrun [pulsar-site]
david-streamlio commented on code in PR #877: URL: https://github.com/apache/pulsar-site/pull/877#discussion_r1548651574 ## docs/io-cli.md: ## @@ -326,9 +326,9 @@ pulsar-admin sinks create options |Flag|Description| ||---| -| `-a`, `--archive` | The path to the archive file for the sink. It also supports url-path (http/https/file [file protocol assumes that file already exists on worker host]) from which worker can download the package. +| `-a`, `--archive` | The absolute path to the archive file for the sink. It also supports url-path (http/https/file [file protocol assumes that file already exists on worker host]) from which worker can download the package. | `--auto-ack` | Whether or not the framework will automatically acknowledge messages. -| `--classname` | The sink's class name if `archive` is file-url-path (file://). +| `--classname` | The sink's class name. Mandatory if `archive` doesn't reference a NAR file. Review Comment: Same ## docs/admin-api-functions.md: ## @@ -46,7 +46,7 @@ pulsar-admin functions create \ --inputs test-input-topic \ --output persistent://public/default/test-output-topic \ --classname org.apache.pulsar.functions.api.examples.ExclamationFunction \ ---jar /examples/api-examples.jar +--jar $PWD/examples/api-examples.jar Review Comment: Is $PWD a safe assumption as to the proper location of the artifact, or should we use a more explicit nomenclature such as `path/to/examples/api-examples.jar` ? ## docs/io-cli.md: ## @@ -245,9 +245,9 @@ pulsar-admin sources localrun options |Flag|Description| ||---| -| `-a`, `--archive` | The path to the NAR archive for the Source. It also supports url-path (http/https/file [file protocol assumes that file already exists on worker host]) from which worker can download the package. +| `-a`, `--archive` | The absolute path to the NAR archive for the Source. It also supports url-path (http/https/file [file protocol assumes that file already exists on worker host]) from which worker can download the package. | `--broker-service-url` | The URL for the Pulsar broker. -|`--classname`|The source's class name if `archive` is file-url-path (file://). +|`--classname`|The source's class name. Mandatory if `archive` doesn't reference a NAR file. Review Comment: Same ## docs/io-cli.md: ## @@ -81,8 +81,8 @@ pulsar-admin sources update options |Flag|Description| ||---| -| `-a`, `--archive` | The path to the NAR archive for the source. It also supports url-path (http/https/file [file protocol assumes that file already exists on worker host]) from which worker can download the package. -| `--classname` | The source's class name if `archive` is file-url-path (file://). +| `-a`, `--archive` | The absolute path to the NAR archive for the source. It also supports url-path (http/https/file [file protocol assumes that file already exists on worker host]) from which worker can download the package. +| `--classname` | The source's class name. Mandatory if `archive` doesn't reference a NAR file. Review Comment: Same as above ## docs/io-cli.md: ## @@ -49,8 +49,8 @@ pulsar-admin sources create options |Flag|Description| ||---| -| `-a`, `--archive` | The path to the NAR archive for the source. It also supports url-path (http/https/file [file protocol assumes that file already exists on worker host]) from which worker can download the package. -| `--classname` | The source's class name if `archive` is file-url-path (file://). +| `-a`, `--archive` | The absolute path to the NAR archive for the source. It also supports url-path (http/https/file [file protocol assumes that file already exists on worker host]) from which worker can download the package. +| `--classname` | The source's class name. Mandatory if `archive` doesn't reference a NAR file. Review Comment: I believe the exclusion for "if `archive` doesn't reference a NAR file." only applies if the NAR file has a proper manifest entry, which may or may not be true. For now, I would recommend leaving it as "Mandatory" without any exclusions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] NumberFormatException when consuming messages from pulsar-admin CLI [pulsar]
lhotari commented on issue #22405: URL: https://github.com/apache/pulsar/issues/22405#issuecomment-2033133048 The impact of the DNS settings being broken in the Dockerfile since Pulsar 3.0.0 is that the positive DNS cache TTL has been 30 seconds. The negative DNS cache TTL has been 0, so there hasn't been any caching at all. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] NumberFormatException when consuming messages from pulsar-admin CLI [pulsar]
lhotari commented on issue #22405: URL: https://github.com/apache/pulsar/issues/22405#issuecomment-2033109881 A possible workaround is to add `-Dsun.net.inetaddr.ttl=1 -Dsun.net.inetaddr.negative.ttl=1` to `OPTS`. For example: ``` export OPTS="-Dsun.net.inetaddr.ttl=1 -Dsun.net.inetaddr.negative.ttl=1" ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][broker] Skip topic.close during unloading if the topic future fails with ownership check, and fix isBundleOwnedByAnyBroker to use ns.checkOwnershipPresentAsync for ExtensibleLoadBalance
github-actions[bot] commented on PR #22406: URL: https://github.com/apache/pulsar/pull/22406#issuecomment-2033086951 @heesung-sn Please add the following content to your PR description and select a checkbox: ``` - [ ] `doc` - [ ] `doc-required` - [ ] `doc-not-needed` - [ ] `doc-complete` ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [fix][broker] Skip topic.close during unloading if the topic future fails with ownership check, and fix isBundleOwnedByAnyBroker to use ns.checkOwnershipPresentAsync for ExtensibleLoadBalancer (#
heesung-sn opened a new pull request, #22406: URL: https://github.com/apache/pulsar/pull/22406 (cherry picked from commit 3eb3b1cd23d2cc11424bf882e244d3bc2e92bf27) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [improve][misc] Remove the call to sun InetAddressCachePolicy [pulsar]
lhotari commented on PR #22329: URL: https://github.com/apache/pulsar/pull/22329#issuecomment-2033087626 It was great that this change was made. It revealed a long time bug in the Dockerfile used in Pulsar 3.0.x, 3.1.x and 3.2.x versions. More explanation in comment https://github.com/apache/pulsar/pull/22400#issuecomment-2033073595 . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] NumberFormatException when consuming messages from pulsar-admin CLI [pulsar]
lhotari commented on issue #22405: URL: https://github.com/apache/pulsar/issues/22405#issuecomment-2033081414 Closing this issue since the fix PR has been merged to branch-3.0 and branch-3.2 . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] NumberFormatException when consuming messages from pulsar-admin CLI [pulsar]
lhotari closed issue #22405: [Bug] NumberFormatException when consuming messages from pulsar-admin CLI URL: https://github.com/apache/pulsar/issues/22405 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] NumberFormatException when consuming messages from pulsar-admin CLI [pulsar]
lhotari commented on issue #22405: URL: https://github.com/apache/pulsar/issues/22405#issuecomment-2033079513 The DNS settings problem has existed for a very long time in the Dockerfile, since #17733 in Pulsar 3.0.0, but it only became visible after #22329 changes. Unfortunately, nobody ran extensive tests with the docker images during the release candidate phase of Pulsar 3.2.2 and Pulsar 3.0.4 . We'll have the fix in Pulsar 3.2.3 and Pulsar 3.0.5 . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][build] Fix networkaddress.cache.negative.ttl config [pulsar]
lhotari commented on PR #22400: URL: https://github.com/apache/pulsar/pull/22400#issuecomment-2033073595 > #21207 introduces the `networkaddress.cache.negative.ttl config` config, which also introduces a bug that appends the "\" at the end: The problem existed before that, but in a different way. See https://github.com/apache/pulsar/blame/55788734a53acd505d97e6f8b3b4bd567603d9ce/docker/pulsar/Dockerfile#L81 . In #21207, the same pattern was replicated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch branch-3.1 updated: [fix][build] Fix networkaddress.cache.negative.ttl config (#22400)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new 20d35f130b8 [fix][build] Fix networkaddress.cache.negative.ttl config
(#22400)
20d35f130b8 is described below
commit 20d35f130b86c204edafb60f18df7f4245159f38
Author: Zixuan Liu
AuthorDate: Wed Apr 3 04:37:27 2024 +0800
[fix][build] Fix networkaddress.cache.negative.ttl config (#22400)
Signed-off-by: Zixuan Liu
(cherry picked from commit be012c8accf58c616c2e66333d5bd3d684326d22)
---
docker/pulsar/Dockerfile | 1 +
1 file changed, 1 insertion(+)
diff --git a/docker/pulsar/Dockerfile b/docker/pulsar/Dockerfile
index b465bbf251d..10bd2b5ee26 100644
--- a/docker/pulsar/Dockerfile
+++ b/docker/pulsar/Dockerfile
@@ -79,6 +79,7 @@ RUN mkdir -p /etc/apt/keyrings \
&& apt-get -y install temurin-17-jdk \
&& export ARCH=$(uname -m | sed -r 's/aarch64/arm64/g' | awk
'!/arm64/{$0="amd64"}1') \
&& echo networkaddress.cache.ttl=1 >>
/usr/lib/jvm/temurin-17-jdk-$ARCH/conf/security/java.security \
+ && echo networkaddress.cache.negative.ttl=1 >>
/usr/lib/jvm/temurin-17-jdk-$ARCH/conf/security/java.security
# Cleanup apt
RUN apt-get -y --purge autoremove \
(pulsar) branch branch-3.2 updated: [fix][build] Fix networkaddress.cache.negative.ttl config (#22400)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 465774ed9bf [fix][build] Fix networkaddress.cache.negative.ttl config
(#22400)
465774ed9bf is described below
commit 465774ed9bfb31cf2f24089ffcf3bf15c5d46e03
Author: Zixuan Liu
AuthorDate: Wed Apr 3 04:37:27 2024 +0800
[fix][build] Fix networkaddress.cache.negative.ttl config (#22400)
Signed-off-by: Zixuan Liu
(cherry picked from commit be012c8accf58c616c2e66333d5bd3d684326d22)
---
docker/pulsar/Dockerfile | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docker/pulsar/Dockerfile b/docker/pulsar/Dockerfile
index 6a0dc0100e7..0bb7c4d539a 100644
--- a/docker/pulsar/Dockerfile
+++ b/docker/pulsar/Dockerfile
@@ -76,7 +76,7 @@ RUN mkdir -p /etc/apt/keyrings \
&& apt-get -y install temurin-${JDK_MAJOR_VERSION:-17}-jdk \
&& export ARCH=$(uname -m | sed -r 's/aarch64/arm64/g' | awk
'!/arm64/{$0="amd64"}1') \
&& echo networkaddress.cache.ttl=1 >>
/usr/lib/jvm/temurin-${JDK_MAJOR_VERSION:-17}-jdk-$ARCH/conf/security/java.security
\
- && echo networkaddress.cache.negative.ttl=1 >>
/usr/lib/jvm/temurin-${JDK_MAJOR_VERSION:-17}-jdk-$ARCH/conf/security/java.security
\
+ && echo networkaddress.cache.negative.ttl=1 >>
/usr/lib/jvm/temurin-${JDK_MAJOR_VERSION:-17}-jdk-$ARCH/conf/security/java.security
# Cleanup apt
RUN apt-get -y --purge autoremove \
Re: [I] [Bug] NumberFormatException when consuming messages from pulsar-admin CLI [pulsar]
lhotari commented on issue #22405: URL: https://github.com/apache/pulsar/issues/22405#issuecomment-2033048482 Fixed by #22400 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch branch-3.0 updated: [fix][build] Fix networkaddress.cache.negative.ttl config (#22400)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new be012c8accf [fix][build] Fix networkaddress.cache.negative.ttl config
(#22400)
be012c8accf is described below
commit be012c8accf58c616c2e66333d5bd3d684326d22
Author: Zixuan Liu
AuthorDate: Wed Apr 3 04:37:27 2024 +0800
[fix][build] Fix networkaddress.cache.negative.ttl config (#22400)
Signed-off-by: Zixuan Liu
---
docker/pulsar/Dockerfile | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docker/pulsar/Dockerfile b/docker/pulsar/Dockerfile
index a4278ad1a71..10bd2b5ee26 100644
--- a/docker/pulsar/Dockerfile
+++ b/docker/pulsar/Dockerfile
@@ -79,7 +79,7 @@ RUN mkdir -p /etc/apt/keyrings \
&& apt-get -y install temurin-17-jdk \
&& export ARCH=$(uname -m | sed -r 's/aarch64/arm64/g' | awk
'!/arm64/{$0="amd64"}1') \
&& echo networkaddress.cache.ttl=1 >>
/usr/lib/jvm/temurin-17-jdk-$ARCH/conf/security/java.security \
- && echo networkaddress.cache.negative.ttl=1 >>
/usr/lib/jvm/temurin-17-jdk-$ARCH/conf/security/java.security \
+ && echo networkaddress.cache.negative.ttl=1 >>
/usr/lib/jvm/temurin-17-jdk-$ARCH/conf/security/java.security
# Cleanup apt
RUN apt-get -y --purge autoremove \
Re: [PR] [fix][build] Fix networkaddress.cache.negative.ttl config [pulsar]
lhotari merged PR #22400: URL: https://github.com/apache/pulsar/pull/22400 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][broker] upgrade jclouds 2.5.0 -> 2.6.0 [pulsar]
pgier commented on PR #0: URL: https://github.com/apache/pulsar/pull/0#issuecomment-2033018107 @lhotari Thanks for fixing this. We should be able to test it, but maybe not for a couple more weeks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] NumberFormatException when consuming messages from pulsar-admin CLI [pulsar]
dao-jun commented on issue #22405: URL: https://github.com/apache/pulsar/issues/22405#issuecomment-2032968849 @nodece -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] [Bug] Force Delete subscription not working on persistent topics [pulsar]
alexesom opened a new issue, #22404: URL: https://github.com/apache/pulsar/issues/22404 ### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Read release policy - [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker. ### Version OS: Ubuntu 23.10 Docker Pulsar standalone 3.1.2 ### Minimal reproduce step Using Java client suggesting that `adminClient` is of type `PulsarAdmin`: invoke `adminClient.topics().deleteSubscription(*topicName*, *subName*, true)` which deletes the subscription forcefully on a persistent topic (on a non-persistent topic it works just fine) and on subscription (same for durable and non-durable) with some active consumers. It could be connected with #9477 also. ### What did you expect to see? Expected to delete subscriptions with active consumers. ### What did you see instead? I get an error: ```org.apache.pulsar.client.admin.PulsarAdminException$PreconditionFailedException: Subscription has active connected consumers``` ### Anything else? _No response_ ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] [Bug] [pulsar]
david-streamlio opened a new issue, #22405: URL: https://github.com/apache/pulsar/issues/22405 ### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Read release policy - [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker. ### Version image: apachepulsar/pulsar-all:3.2.2 platform: linux/amd64 ### Minimal reproduce step ./bin/pulsar-client consume -n 0 -p Earliest -s sub ### What did you expect to see? No errors ### What did you see instead? $ ./bin/pulsar-client consume -n 0 -p Earliest -s sub persistent://feeds/realtime/coinbase-rfq 2024-04-02T18:34:44,234+ [main] WARN org.apache.pulsar.common.util.netty.DnsResolverUtil - Cannot get DNS TTL settings java.lang.NumberFormatException: For input string: "1 RUN apt-get -y --purge autoremove" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:67) ~[?:?] at java.lang.Integer.parseInt(Integer.java:668) ~[?:?] at java.lang.Integer.valueOf(Integer.java:973) ~[?:?] at java.lang.Integer.decode(Integer.java:1458) ~[?:?] at java.util.Optional.map(Optional.java:260) ~[?:?] at org.apache.pulsar.common.util.netty.DnsResolverUtil.(DnsResolverUtil.java:71) ~[org.apache.pulsar-pulsar-common-3.2.2.jar:3.2.2] at org.apache.pulsar.client.impl.ConnectionPool.createAddressResolver(ConnectionPool.java:164) ~[org.apache.pulsar-pulsar-client-original-3.2.2.jar:3.2.2] at org.apache.pulsar.client.impl.ConnectionPool.lambda$new$1(ConnectionPool.java:127) ~[org.apache.pulsar-pulsar-client-original-3.2.2.jar:3.2.2] at java.util.Optional.orElseGet(Optional.java:364) ~[?:?] at org.apache.pulsar.client.impl.ConnectionPool.(ConnectionPool.java:127) ~[org.apache.pulsar-pulsar-client-original-3.2.2.jar:3.2.2] at org.apache.pulsar.client.impl.ConnectionPool.(ConnectionPool.java:96) ~[org.apache.pulsar-pulsar-client-original-3.2.2.jar:3.2.2] at org.apache.pulsar.client.impl.ConnectionPool.(ConnectionPool.java:91) ~[org.apache.pulsar-pulsar-client-original-3.2.2.jar:3.2.2] at org.apache.pulsar.client.impl.PulsarClientImpl.(PulsarClientImpl.java:199) ~[org.apache.pulsar-pulsar-client-original-3.2.2.jar:3.2.2] at org.apache.pulsar.client.impl.PulsarClientImpl.(PulsarClientImpl.java:156) ~[org.apache.pulsar-pulsar-client-original-3.2.2.jar:3.2.2] at org.apache.pulsar.client.impl.ClientBuilderImpl.build(ClientBuilderImpl.java:65) ~[org.apache.pulsar-pulsar-client-original-3.2.2.jar:3.2.2] at org.apache.pulsar.client.cli.CmdConsume.consume(CmdConsume.java:149) ~[org.apache.pulsar-pulsar-client-tools-3.2.2.jar:3.2.2] at org.apache.pulsar.client.cli.CmdConsume.run(CmdConsume.java:141) ~[org.apache.pulsar-pulsar-client-tools-3.2.2.jar:3.2.2] at org.apache.pulsar.client.cli.PulsarClientTool.run(PulsarClientTool.java:246) ~[org.apache.pulsar-pulsar-client-tools-3.2.2.jar:3.2.2] at org.apache.pulsar.client.cli.PulsarClientTool.main(PulsarClientTool.java:286) ~[org.apache.pulsar-pulsar-client-tools-3.2.2.jar:3.2.2] ### Anything else? _No response_ ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [fix][broker] Skip topic.close during unloading if the topic future fails with ownership check, and fix isBundleOwnedByAnyBroker to use ns.checkOwnershipPresentAsync for ExtensibleLoadBalancer (#
heesung-sn opened a new pull request, #22403: URL: https://github.com/apache/pulsar/pull/22403 (cherry picked from commit 3eb3b1cd23d2cc11424bf882e244d3bc2e92bf27) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][broker] Skip topic.close during unloading if the topic future fails with ownership check, and fix isBundleOwnedByAnyBroker to use ns.checkOwnershipPresentAsync for ExtensibleLoadBalance
github-actions[bot] commented on PR #22403: URL: https://github.com/apache/pulsar/pull/22403#issuecomment-2032767571 @heesung-sn Please add the following content to your PR description and select a checkbox: ``` - [ ] `doc` - [ ] `doc-required` - [ ] `doc-not-needed` - [ ] `doc-complete` ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] [Bug] Unable to update subscription properties on non-durable subscription [pulsar]
alexesom opened a new issue, #22402: URL: https://github.com/apache/pulsar/issues/22402 ### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Read release policy - [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker. ### Version OS: Ubuntu 23.10 Docker Pulsar standalone 3.1.2 ### Minimal reproduce step Using Java client suggesting that `adminClient` is of type `PulsarAdmin`: invoke `adminClient.topics().updateSubscriptionProperties(*topicName*, *subName*, *props*)` on a persistent topic (on a non-persistent topic it works just fine). ### What did you expect to see? Expected to update properties for subscription on a topic ### What did you see instead? I got error: ``` --- An unexpected error occurred in the server --- Message: --- An unexpected error occurred in the server --- Message: Cannot invoke "org.apache.bookkeeper.mledger.proto.MLDataFormats$ManagedCursorInfo.hasCursorsLedgerId()" because "other" is null Stacktrace: java.lang.NullPointerException: Cannot invoke "org.apache.bookkeeper.mledger.proto.MLDataFormats$ManagedCursorInfo.hasCursorsLedgerId()" because "other" is null ``` ### Anything else? _No response_ ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Changed the aggregatePublisherStatsByProducerName config default to true [pulsar]
heesung-sn closed pull request #19114: Changed the aggregatePublisherStatsByProducerName config default to true URL: https://github.com/apache/pulsar/pull/19114 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [improve][broker] Added property map in AuthenticationDataSourceHttp [pulsar]
heesung-sn closed pull request #21547: [improve][broker] Added property map in AuthenticationDataSourceHttp URL: https://github.com/apache/pulsar/pull/21547 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][client] Fix getProxyConnection when the topic is migrated [pulsar]
heesung-sn closed pull request #22085: [fix][client] Fix getProxyConnection when the topic is migrated URL: https://github.com/apache/pulsar/pull/22085 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][broker] Update TransferShedder underloaded broker check to consider max loaded broker's msgThroughputEMA and update IsExtensibleLoadBalancerImpl check [pulsar]
heesung-sn closed pull request #22321: [fix][broker] Update TransferShedder underloaded broker check to consider max loaded broker's msgThroughputEMA and update IsExtensibleLoadBalancerImpl check URL: https://github.com/apache/pulsar/pull/22321 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [feat][site] PIP-264 Add OpenTelemetry metrics instructions [pulsar-site]
dragosvictor commented on code in PR #841: URL: https://github.com/apache/pulsar-site/pull/841#discussion_r1548265079 ## docs/deploy-monitoring.md: ## @@ -117,3 +117,119 @@ The following are some Grafana dashboards examples: ## Alerting rules You can set alerting rules according to your Pulsar environment. To configure alerting rules for Apache Pulsar, refer to [alerting rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/). + +## OpenTelemetry + +### Status +Pulsar emits OpenTelemetry metrics starting from version 3.3.0. OpenTelemetry log and trace signals are not exposed by +Pulsar. OpenTelemetry support is currently **experimental** and complements the pre-existing Prometheus metric system, +with the goal of eventually replacing it. The metrics it exposes are semantically equivalent to the Prometheus metrics. + +For a detailed list of OpenTelemetry metrics exposed by Pulsar, refer to [OpenTelemetry Metrics](reference-metrics-opentelemetry.md). + +### Scope +Pulsar OpenTelemetry metrics are gradually being added for the broker only. Support for the proxy and function worker is +planned for a future release. + +### OpenTelemetry Configuration +Pulsar natively supports OpenTelemetry via manual instrumentation, instead of relying on the OpenTelemetry automatic +instrumentation agent. Pulsar uses the auto-configuration [extension](https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk-extensions/autoconfigure/README.md) +of OpenTelemetry to manage the SDK configuration. The extension allows parameter input from environment variables and +Java system properties. The instructions below rely on environment variables, but can be adapted to use system +properties too. These variables must be exposed to the Pulsar process via the respective deployment method. + +Note that the experimental [file based configuration](https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk-extensions/autoconfigure/README.md#file-configuration) +is not currently supported by Pulsar. + + Telemetry Enablement +The experimental OpenTelemetry feature is explicitly disabled by default in Pulsar. Set environment variable +`OTEL_SDK_DISABLED=false` to enable the SDK. When disabled, metrics will not be collected nor exported. + + Exporter Configuration + +Exporters using the native OpenTelemetry Protocol and Prometheus are included in the Pulsar distribution assembly by +default and can be used out-of-the-box. Other exporters are not currently supported. + Review Comment: Done! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][client] client side memory leak when call ProducerBase.sendAsync(Message) [pulsar]
poorbarcode commented on PR #22393: URL: https://github.com/apache/pulsar/pull/22393#issuecomment-2032515873 > @poorbarcode I just noticed there are some PRs are fixing WARN/ERROR logs recently, maybe it's better to consider it carefully. ReferenceCountUtil.safeRelease(payload) will do nothing if the payload is `null`, I added a error log manually. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][client] client side memory leak when call ProducerBase.sendAsync(Message) [pulsar]
poorbarcode commented on code in PR #22393:
URL: https://github.com/apache/pulsar/pull/22393#discussion_r1548195395
##
pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java:
##
@@ -378,80 +378,93 @@ CompletableFuture internalSendAsync(Message
message) {
pendingMessagesUpDownCounter.increment();
pendingBytesUpDownCounter.add(msgSize);
-sendAsync(interceptorMessage, new SendCallback() {
-SendCallback nextCallback = null;
-MessageImpl nextMsg = null;
-long createdAt = System.nanoTime();
+sendAsync(interceptorMessage, new DefaultSendMessageCallback(future,
interceptorMessage, msgSize));
+return future;
+}
-@Override
-public CompletableFuture getFuture() {
-return future;
-}
+private class DefaultSendMessageCallback implements SendCallback {
-@Override
-public SendCallback getNextSendCallback() {
-return nextCallback;
-}
+CompletableFuture sendFuture;
+MessageImpl currentMsg;
+int msgSize;
+long createdAt = System.nanoTime();
+SendCallback nextCallback = null;
+MessageImpl nextMsg = null;
-@Override
-public MessageImpl getNextMessage() {
-return nextMsg;
-}
+DefaultSendMessageCallback(CompletableFuture sendFuture,
MessageImpl currentMsg, int msgSize) {
+this.sendFuture = sendFuture;
+this.currentMsg = currentMsg;
+this.msgSize = msgSize;
+}
-@Override
-public void sendComplete(Exception e) {
-long latencyNanos = System.nanoTime() - createdAt;
-pendingMessagesUpDownCounter.decrement();
-pendingBytesUpDownCounter.subtract(msgSize);
+@Override
+public CompletableFuture getFuture() {
+return sendFuture;
+}
+
+@Override
+public SendCallback getNextSendCallback() {
+return nextCallback;
+}
+
+@Override
+public MessageImpl getNextMessage() {
+return nextMsg;
+}
+@Override
+public void sendComplete(Exception e) {
+SendCallback loopingCallback = this;
+MessageImpl loopingMsg = currentMsg;
+while (loopingCallback != null) {
+onSendComplete(e, loopingCallback, loopingMsg);
+loopingMsg = loopingCallback.getNextMessage();
+loopingCallback = loopingCallback.getNextSendCallback();
+}
+}
+
+private void onSendComplete(Exception e, SendCallback sendCallback,
MessageImpl msg) {
+long createdAt = (sendCallback instanceof
ProducerImpl.DefaultSendMessageCallback)
+? ((DefaultSendMessageCallback) sendCallback).createdAt :
this.createdAt;
+long latencyNanos = System.nanoTime() - createdAt;
+pendingMessagesUpDownCounter.decrement();
+pendingBytesUpDownCounter.subtract(msgSize);
+ByteBuf payload = msg.getDataBuffer();
+if (e != null) {
+latencyHistogram.recordFailure(latencyNanos);
+stats.incrementSendFailed();
try {
-if (e != null) {
-latencyHistogram.recordFailure(latencyNanos);
-stats.incrementSendFailed();
-onSendAcknowledgement(interceptorMessage, null, e);
-future.completeExceptionally(e);
-} else {
-latencyHistogram.recordSuccess(latencyNanos);
-publishedBytesCounter.add(msgSize);
-onSendAcknowledgement(interceptorMessage,
interceptorMessage.getMessageId(), null);
-future.complete(interceptorMessage.getMessageId());
-stats.incrementNumAcksReceived(latencyNanos);
-}
+onSendAcknowledgement(msg, null, e);
+sendCallback.getFuture().completeExceptionally(e);
} finally {
-interceptorMessage.getDataBuffer().release();
+if (payload == null) {
+log.error("[{}] [{}] Payload is null when calling a
failed onSendComplete, which is not"
++ " expected.", topic, producerName);
Review Comment:
Added a `return`. `ReferenceCountUtil.safeRelease(payload)` will do nothing
if the `payload` is null
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure
Re: [PR] Bump io.vertx:vertx-core from 4.3.8 to 4.5.3 [pulsar]
github-actions[bot] commented on PR #22401: URL: https://github.com/apache/pulsar/pull/22401#issuecomment-2032492707 @dependabot[bot] Please add the following content to your PR description and select a checkbox: ``` - [ ] `doc` - [ ] `doc-required` - [ ] `doc-not-needed` - [ ] `doc-complete` ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch dependabot/maven/io.vertx-vertx-core-4.5.3 created (now f0460c9fb0e)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/io.vertx-vertx-core-4.5.3 in repository https://gitbox.apache.org/repos/asf/pulsar.git at f0460c9fb0e Bump io.vertx:vertx-core from 4.3.8 to 4.5.3 No new revisions were added by this update.
[PR] Bump io.vertx:vertx-core from 4.3.8 to 4.5.3 [pulsar]
dependabot[bot] opened a new pull request, #22401: URL: https://github.com/apache/pulsar/pull/22401 Bumps [io.vertx:vertx-core](https://github.com/eclipse/vert.x) from 4.3.8 to 4.5.3. Commits https://github.com/eclipse-vertx/vert.x/commit/ce654726ce4c7642143258c78b3f8992be99e3c7;>ce65472 Releasing 4.5.3 https://github.com/eclipse-vertx/vert.x/commit/7ad34ea9d78f85e26b231ee3ec8d492d10046479;>7ad34ea The SslChannelProvider class maintains a map of server name to Netty SslConte... https://github.com/eclipse-vertx/vert.x/commit/ea749f678a429a97c10d4207104c469c6a856561;>ea749f6 Refactor and unify client/server code in SslChannelProvider/SslContextProvider https://github.com/eclipse-vertx/vert.x/commit/1487999acadc5beff4590c4fefd70a169515c440;>1487999 The TCP client will not send the correct server name to the client due to SSL... https://github.com/eclipse-vertx/vert.x/commit/3e1096f9adc05f24947fbab8aa99606aa8506142;>3e1096f Add helper method to determine if request can be upgraded to websocket. https://github.com/eclipse-vertx/vert.x/commit/6ddadec7a53e5898d2d40d578749d35dbca65f63;>6ddadec Allow dynamic update of read and write limits set for Servers https://github.com/eclipse-vertx/vert.x/commit/3ab73e0c3ace11abf7d58cedbf030f332d01bcfd;>3ab73e0 Merge pull request https://redirect.github.com/eclipse/vert.x/issues/5094;>#5094 from tsegismont/no-pretty-mapper-internally https://github.com/eclipse-vertx/vert.x/commit/1da42d42a9c24a8a87402bcb5937b202212f317c;>1da42d4 Clarify expectations in testObjectMapperConfigAppliesToPrettyPrinting https://github.com/eclipse-vertx/vert.x/commit/87aaa562066c1127c649a1b6ebf2b86007730f13;>87aaa56 Load pretty mapper eagerly https://github.com/eclipse-vertx/vert.x/commit/563912dc1e182809442134142134ab07fe85eade;>563912d ObjectMapper configuration should apply when pretty printing Additional commits viewable in https://github.com/eclipse/vert.x/compare/4.3.8...4.5.3;>compare view [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/pulsar/network/alerts). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][build] Fix networkaddress.cache.negative.ttl config [pulsar]
nodece commented on PR #22400: URL: https://github.com/apache/pulsar/pull/22400#issuecomment-2032478584 /pulsarbot rerun-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [fix][build] Fix networkaddress.cache.negative.ttl config [pulsar]
nodece opened a new pull request, #22400: URL: https://github.com/apache/pulsar/pull/22400 ### Motivation https://github.com/apache/pulsar/pull/21207 introduces the `networkaddress.cache.negative.ttl config` config, which also introduces a bug that appends the "\\" at the end: ``` 2024-04-02T15:00:10,494+ [pulsar-perf-producer-exec-1-1] INFO org.apache.pulsar.testclient.PerformanceProducer - Started performance test thread 0 2024-04-02T15:00:10,671+ [pulsar-perf-producer-exec-1-1] WARN org.apache.pulsar.common.util.netty.DnsResolverUtil - Cannot get DNS TTL settings java.lang.NumberFormatException: For input string: "1 RUN apt-get -y --purge autoremove" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:67) ~[?:?] at java.lang.Integer.parseInt(Integer.java:668) ~[?:?] at java.lang.Integer.valueOf(Integer.java:973) ~[?:?] at java.lang.Integer.decode(Integer.java:1458) ~[?:?] at java.util.Optional.map(Optional.java:260) ~[?:?] at org.apache.pulsar.common.util.netty.DnsResolverUtil.(DnsResolverUtil.java:71) ~[org.apache.pulsar-pulsar-common-3.2.2.jar:3.2.2] at org.apache.pulsar.client.impl.ConnectionPool.createAddressResolver(ConnectionPool.java:164) ~[org.apache.pulsar-pulsar-client-original-3.2.2.jar:3.2.2] at org.apache.pulsar.client.impl.ConnectionPool.lambda$new$1(ConnectionPool.java:127) ~[org.apache.pulsar-pulsar-client-original-3.2.2.jar:3.2.2] at java.util.Optional.orElseGet(Optional.java:364) ~[?:?] at org.apache.pulsar.client.impl.ConnectionPool.(ConnectionPool.java:127) ~[org.apache.pulsar-pulsar-client-original-3.2.2.jar:3.2.2] at org.apache.pulsar.client.impl.ConnectionPool.(ConnectionPool.java:96) ~[org.apache.pulsar-pulsar-client-original-3.2.2.jar:3.2.2] at org.apache.pulsar.client.impl.ConnectionPool.(ConnectionPool.java:91) ~[org.apache.pulsar-pulsar-client-original-3.2.2.jar:3.2.2] at org.apache.pulsar.client.impl.PulsarClientImpl.(PulsarClientImpl.java:199) ~[org.apache.pulsar-pulsar-client-original-3.2.2.jar:3.2.2] at org.apache.pulsar.client.impl.PulsarClientImpl.(PulsarClientImpl.java:156) ~[org.apache.pulsar-pulsar-client-original-3.2.2.jar:3.2.2] at org.apache.pulsar.client.impl.ClientBuilderImpl.build(ClientBuilderImpl.java:65) ~[org.apache.pulsar-pulsar-client-original-3.2.2.jar:3.2.2] at org.apache.pulsar.testclient.PerformanceProducer.runProducer(PerformanceProducer.java:519) ~[org.apache.pulsar-pulsar-testclient-3.2.2.jar:3.2.2] at org.apache.pulsar.testclient.PerformanceProducer.lambda$main$1(PerformanceProducer.java:351) ~[org.apache.pulsar-pulsar-testclient-3.2.2.jar:3.2.2] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.105.Final.jar:4.1.105.Final] at java.lang.Thread.run(Thread.java:840) ~[?:?] 2024-04-02T15:00:10,763+ [pulsar-perf-producer-exec-1-1] INFO org.apache.pulsar.testclient.PerformanceProducer - Adding 1 publishers on topic my-tenant/my-namespace/rg-test-1 ``` ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch branch-3.2 updated: [improve][broker] Don't log brokerClientAuthenticationParameters and bookkeeperClientAuthenticationParameters by default (#22395)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 086be1038a5 [improve][broker] Don't log
brokerClientAuthenticationParameters and
bookkeeperClientAuthenticationParameters by default (#22395)
086be1038a5 is described below
commit 086be1038a58d8036b7b8823b9731514c55967d3
Author: Lari Hotari
AuthorDate: Tue Apr 2 08:17:26 2024 -0700
[improve][broker] Don't log brokerClientAuthenticationParameters and
bookkeeperClientAuthenticationParameters by default (#22395)
(cherry picked from commit ad28a7c1ef717aafa1c457762f43101152665572)
---
.../src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java| 2 ++
1 file changed, 2 insertions(+)
diff --git
a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
index 4f2d56fc07e..88e5d76738f 100644
---
a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
+++
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
@@ -1683,6 +1683,7 @@ public class ServiceConfiguration implements
PulsarConfiguration {
category = CATEGORY_STORAGE_BK,
doc = "Parameters for bookkeeper auth plugin"
)
+@ToString.Exclude
private String bookkeeperClientAuthenticationParameters;
@FieldContext(
@@ -3282,6 +3283,7 @@ public class ServiceConfiguration implements
PulsarConfiguration {
doc = "Authentication parameters of the authentication plugin the
broker is using to connect "
+ "to other brokers"
)
+@ToString.Exclude
private String brokerClientAuthenticationParameters = "";
@FieldContext(
category = CATEGORY_REPLICATION,
(pulsar) branch branch-3.1 updated: [improve][broker] Don't log brokerClientAuthenticationParameters and bookkeeperClientAuthenticationParameters by default (#22395)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new 3392170bfb5 [improve][broker] Don't log
brokerClientAuthenticationParameters and
bookkeeperClientAuthenticationParameters by default (#22395)
3392170bfb5 is described below
commit 3392170bfb51497ac5026264ed34513c37dc969d
Author: Lari Hotari
AuthorDate: Tue Apr 2 08:17:26 2024 -0700
[improve][broker] Don't log brokerClientAuthenticationParameters and
bookkeeperClientAuthenticationParameters by default (#22395)
(cherry picked from commit ad28a7c1ef717aafa1c457762f43101152665572)
---
.../src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java| 2 ++
1 file changed, 2 insertions(+)
diff --git
a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
index 6175835826e..be8ea2a851b 100644
---
a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
+++
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
@@ -1689,6 +1689,7 @@ public class ServiceConfiguration implements
PulsarConfiguration {
category = CATEGORY_STORAGE_BK,
doc = "Parameters for bookkeeper auth plugin"
)
+@ToString.Exclude
private String bookkeeperClientAuthenticationParameters;
@FieldContext(
@@ -3259,6 +3260,7 @@ public class ServiceConfiguration implements
PulsarConfiguration {
doc = "Authentication parameters of the authentication plugin the
broker is using to connect "
+ "to other brokers"
)
+@ToString.Exclude
private String brokerClientAuthenticationParameters = "";
@FieldContext(
category = CATEGORY_REPLICATION,
(pulsar) branch branch-3.0 updated: [improve][broker] Don't log brokerClientAuthenticationParameters and bookkeeperClientAuthenticationParameters by default (#22395)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new e6501bddc95 [improve][broker] Don't log
brokerClientAuthenticationParameters and
bookkeeperClientAuthenticationParameters by default (#22395)
e6501bddc95 is described below
commit e6501bddc9546635623955a90155c5de97571083
Author: Lari Hotari
AuthorDate: Tue Apr 2 08:17:26 2024 -0700
[improve][broker] Don't log brokerClientAuthenticationParameters and
bookkeeperClientAuthenticationParameters by default (#22395)
(cherry picked from commit ad28a7c1ef717aafa1c457762f43101152665572)
---
.../src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java| 2 ++
1 file changed, 2 insertions(+)
diff --git
a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
index 077228191ad..45bbc7d1924 100644
---
a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
+++
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
@@ -1674,6 +1674,7 @@ public class ServiceConfiguration implements
PulsarConfiguration {
category = CATEGORY_STORAGE_BK,
doc = "Parameters for bookkeeper auth plugin"
)
+@ToString.Exclude
private String bookkeeperClientAuthenticationParameters;
@FieldContext(
@@ -3220,6 +3221,7 @@ public class ServiceConfiguration implements
PulsarConfiguration {
doc = "Authentication parameters of the authentication plugin the
broker is using to connect "
+ "to other brokers"
)
+@ToString.Exclude
private String brokerClientAuthenticationParameters = "";
@FieldContext(
category = CATEGORY_REPLICATION,
(pulsar) branch master updated: [improve][broker] Don't log brokerClientAuthenticationParameters and bookkeeperClientAuthenticationParameters by default (#22395)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new ad28a7c1ef7 [improve][broker] Don't log
brokerClientAuthenticationParameters and
bookkeeperClientAuthenticationParameters by default (#22395)
ad28a7c1ef7 is described below
commit ad28a7c1ef717aafa1c457762f43101152665572
Author: Lari Hotari
AuthorDate: Tue Apr 2 08:17:26 2024 -0700
[improve][broker] Don't log brokerClientAuthenticationParameters and
bookkeeperClientAuthenticationParameters by default (#22395)
---
.../src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java| 2 ++
1 file changed, 2 insertions(+)
diff --git
a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
index e088f50a05c..80dfcaf4b0b 100644
---
a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
+++
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
@@ -1689,6 +1689,7 @@ public class ServiceConfiguration implements
PulsarConfiguration {
category = CATEGORY_STORAGE_BK,
doc = "Parameters for bookkeeper auth plugin"
)
+@ToString.Exclude
private String bookkeeperClientAuthenticationParameters;
@FieldContext(
@@ -3303,6 +3304,7 @@ public class ServiceConfiguration implements
PulsarConfiguration {
doc = "Authentication parameters of the authentication plugin the
broker is using to connect "
+ "to other brokers"
)
+@ToString.Exclude
private String brokerClientAuthenticationParameters = "";
@FieldContext(
category = CATEGORY_REPLICATION,
Re: [PR] [improve][broker] Don't log brokerClientAuthenticationParameters and bookkeeperClientAuthenticationParameters by default [pulsar]
lhotari merged PR #22395: URL: https://github.com/apache/pulsar/pull/22395 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][client] client side memory leak when call ProducerBase.sendAsync(Message) [pulsar]
gaoran10 commented on code in PR #22393:
URL: https://github.com/apache/pulsar/pull/22393#discussion_r1548082039
##
pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java:
##
@@ -378,80 +378,93 @@ CompletableFuture internalSendAsync(Message
message) {
pendingMessagesUpDownCounter.increment();
pendingBytesUpDownCounter.add(msgSize);
-sendAsync(interceptorMessage, new SendCallback() {
-SendCallback nextCallback = null;
-MessageImpl nextMsg = null;
-long createdAt = System.nanoTime();
+sendAsync(interceptorMessage, new DefaultSendMessageCallback(future,
interceptorMessage, msgSize));
+return future;
+}
-@Override
-public CompletableFuture getFuture() {
-return future;
-}
+private class DefaultSendMessageCallback implements SendCallback {
-@Override
-public SendCallback getNextSendCallback() {
-return nextCallback;
-}
+CompletableFuture sendFuture;
+MessageImpl currentMsg;
+int msgSize;
+long createdAt = System.nanoTime();
+SendCallback nextCallback = null;
+MessageImpl nextMsg = null;
-@Override
-public MessageImpl getNextMessage() {
-return nextMsg;
-}
+DefaultSendMessageCallback(CompletableFuture sendFuture,
MessageImpl currentMsg, int msgSize) {
+this.sendFuture = sendFuture;
+this.currentMsg = currentMsg;
+this.msgSize = msgSize;
+}
-@Override
-public void sendComplete(Exception e) {
-long latencyNanos = System.nanoTime() - createdAt;
-pendingMessagesUpDownCounter.decrement();
-pendingBytesUpDownCounter.subtract(msgSize);
+@Override
+public CompletableFuture getFuture() {
+return sendFuture;
+}
+
+@Override
+public SendCallback getNextSendCallback() {
+return nextCallback;
+}
+
+@Override
+public MessageImpl getNextMessage() {
+return nextMsg;
+}
+@Override
+public void sendComplete(Exception e) {
+SendCallback loopingCallback = this;
+MessageImpl loopingMsg = currentMsg;
+while (loopingCallback != null) {
+onSendComplete(e, loopingCallback, loopingMsg);
+loopingMsg = loopingCallback.getNextMessage();
+loopingCallback = loopingCallback.getNextSendCallback();
+}
+}
+
+private void onSendComplete(Exception e, SendCallback sendCallback,
MessageImpl msg) {
+long createdAt = (sendCallback instanceof
ProducerImpl.DefaultSendMessageCallback)
+? ((DefaultSendMessageCallback) sendCallback).createdAt :
this.createdAt;
+long latencyNanos = System.nanoTime() - createdAt;
+pendingMessagesUpDownCounter.decrement();
+pendingBytesUpDownCounter.subtract(msgSize);
+ByteBuf payload = msg.getDataBuffer();
+if (e != null) {
+latencyHistogram.recordFailure(latencyNanos);
+stats.incrementSendFailed();
try {
-if (e != null) {
-latencyHistogram.recordFailure(latencyNanos);
-stats.incrementSendFailed();
-onSendAcknowledgement(interceptorMessage, null, e);
-future.completeExceptionally(e);
-} else {
-latencyHistogram.recordSuccess(latencyNanos);
-publishedBytesCounter.add(msgSize);
-onSendAcknowledgement(interceptorMessage,
interceptorMessage.getMessageId(), null);
-future.complete(interceptorMessage.getMessageId());
-stats.incrementNumAcksReceived(latencyNanos);
-}
+onSendAcknowledgement(msg, null, e);
+sendCallback.getFuture().completeExceptionally(e);
} finally {
-interceptorMessage.getDataBuffer().release();
+if (payload == null) {
+log.error("[{}] [{}] Payload is null when calling a
failed onSendComplete, which is not"
++ " expected.", topic, producerName);
Review Comment:
Do we need to return?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [fix][client] client side memory leak when call ProducerBase.sendAsync(Message) [pulsar]
gaoran10 commented on code in PR #22393:
URL: https://github.com/apache/pulsar/pull/22393#discussion_r1548082039
##
pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java:
##
@@ -378,80 +378,93 @@ CompletableFuture internalSendAsync(Message
message) {
pendingMessagesUpDownCounter.increment();
pendingBytesUpDownCounter.add(msgSize);
-sendAsync(interceptorMessage, new SendCallback() {
-SendCallback nextCallback = null;
-MessageImpl nextMsg = null;
-long createdAt = System.nanoTime();
+sendAsync(interceptorMessage, new DefaultSendMessageCallback(future,
interceptorMessage, msgSize));
+return future;
+}
-@Override
-public CompletableFuture getFuture() {
-return future;
-}
+private class DefaultSendMessageCallback implements SendCallback {
-@Override
-public SendCallback getNextSendCallback() {
-return nextCallback;
-}
+CompletableFuture sendFuture;
+MessageImpl currentMsg;
+int msgSize;
+long createdAt = System.nanoTime();
+SendCallback nextCallback = null;
+MessageImpl nextMsg = null;
-@Override
-public MessageImpl getNextMessage() {
-return nextMsg;
-}
+DefaultSendMessageCallback(CompletableFuture sendFuture,
MessageImpl currentMsg, int msgSize) {
+this.sendFuture = sendFuture;
+this.currentMsg = currentMsg;
+this.msgSize = msgSize;
+}
-@Override
-public void sendComplete(Exception e) {
-long latencyNanos = System.nanoTime() - createdAt;
-pendingMessagesUpDownCounter.decrement();
-pendingBytesUpDownCounter.subtract(msgSize);
+@Override
+public CompletableFuture getFuture() {
+return sendFuture;
+}
+
+@Override
+public SendCallback getNextSendCallback() {
+return nextCallback;
+}
+
+@Override
+public MessageImpl getNextMessage() {
+return nextMsg;
+}
+@Override
+public void sendComplete(Exception e) {
+SendCallback loopingCallback = this;
+MessageImpl loopingMsg = currentMsg;
+while (loopingCallback != null) {
+onSendComplete(e, loopingCallback, loopingMsg);
+loopingMsg = loopingCallback.getNextMessage();
+loopingCallback = loopingCallback.getNextSendCallback();
+}
+}
+
+private void onSendComplete(Exception e, SendCallback sendCallback,
MessageImpl msg) {
+long createdAt = (sendCallback instanceof
ProducerImpl.DefaultSendMessageCallback)
+? ((DefaultSendMessageCallback) sendCallback).createdAt :
this.createdAt;
+long latencyNanos = System.nanoTime() - createdAt;
+pendingMessagesUpDownCounter.decrement();
+pendingBytesUpDownCounter.subtract(msgSize);
+ByteBuf payload = msg.getDataBuffer();
+if (e != null) {
+latencyHistogram.recordFailure(latencyNanos);
+stats.incrementSendFailed();
try {
-if (e != null) {
-latencyHistogram.recordFailure(latencyNanos);
-stats.incrementSendFailed();
-onSendAcknowledgement(interceptorMessage, null, e);
-future.completeExceptionally(e);
-} else {
-latencyHistogram.recordSuccess(latencyNanos);
-publishedBytesCounter.add(msgSize);
-onSendAcknowledgement(interceptorMessage,
interceptorMessage.getMessageId(), null);
-future.complete(interceptorMessage.getMessageId());
-stats.incrementNumAcksReceived(latencyNanos);
-}
+onSendAcknowledgement(msg, null, e);
+sendCallback.getFuture().completeExceptionally(e);
} finally {
-interceptorMessage.getDataBuffer().release();
+if (payload == null) {
+log.error("[{}] [{}] Payload is null when calling a
failed onSendComplete, which is not"
++ " expected.", topic, producerName);
Review Comment:
Maybe we should return.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [fix][client] client side memory leak when call ProducerBase.sendAsync(Message) [pulsar]
dao-jun commented on PR #22393: URL: https://github.com/apache/pulsar/pull/22393#issuecomment-2032300390 > @dao-jun > > > double release a ReferentCount Object will throw an exception, and ReferenceCountUtil.safeRelease(...) prints WARN log. > > I think it is ok. If it will be released twice, we need this error log to let us know something is not as expected. @poorbarcode I just noticed there are some PRs are fixing WARN/ERROR logs recently, maybe it's better to consider it carefully. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [feat][site] PIP-264 Add OpenTelemetry metrics instructions [pulsar-site]
asafm commented on code in PR #841: URL: https://github.com/apache/pulsar-site/pull/841#discussion_r1548040083 ## docs/deploy-monitoring.md: ## @@ -117,3 +117,119 @@ The following are some Grafana dashboards examples: ## Alerting rules You can set alerting rules according to your Pulsar environment. To configure alerting rules for Apache Pulsar, refer to [alerting rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/). + +## OpenTelemetry + +### Status +Pulsar emits OpenTelemetry metrics starting from version 3.3.0. OpenTelemetry log and trace signals are not exposed by +Pulsar. OpenTelemetry support is currently **experimental** and complements the pre-existing Prometheus metric system, +with the goal of eventually replacing it. The metrics it exposes are semantically equivalent to the Prometheus metrics. + +For a detailed list of OpenTelemetry metrics exposed by Pulsar, refer to [OpenTelemetry Metrics](reference-metrics-opentelemetry.md). + +### Scope +Pulsar OpenTelemetry metrics are gradually being added for the broker only. Support for the proxy and function worker is +planned for a future release. + +### OpenTelemetry Configuration +Pulsar natively supports OpenTelemetry via manual instrumentation, instead of relying on the OpenTelemetry automatic +instrumentation agent. Pulsar uses the auto-configuration [extension](https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk-extensions/autoconfigure/README.md) +of OpenTelemetry to manage the SDK configuration. The extension allows parameter input from environment variables and +Java system properties. The instructions below rely on environment variables, but can be adapted to use system +properties too. These variables must be exposed to the Pulsar process via the respective deployment method. + +Note that the experimental [file based configuration](https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk-extensions/autoconfigure/README.md#file-configuration) +is not currently supported by Pulsar. + + Telemetry Enablement +The experimental OpenTelemetry feature is explicitly disabled by default in Pulsar. Set environment variable +`OTEL_SDK_DISABLED=false` to enable the SDK. When disabled, metrics will not be collected nor exported. + + Exporter Configuration + +Exporters using the native OpenTelemetry Protocol and Prometheus are included in the Pulsar distribution assembly by +default and can be used out-of-the-box. Other exporters are not currently supported. + Review Comment: Copying from earlier comment which got detached :) We should add here: "The native OTLP exporter is the recommended way to obtain metrics out of Pulsar as the Apache Pulsar community is working on modifying it (and not Prometheus) to be highly performant." -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][client] client side memory leak when call ProducerBase.sendAsync(Message) [pulsar]
poorbarcode commented on code in PR #22393:
URL: https://github.com/apache/pulsar/pull/22393#discussion_r1548017855
##
pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java:
##
@@ -418,16 +419,14 @@ public void sendComplete(Exception e) {
stats.incrementNumAcksReceived(latencyNanos);
}
} finally {
-interceptorMessage.getDataBuffer().release();
+ReferenceCountUtil.safeRelease(payloadInCurrentMsg);
Review Comment:
> Maybe we can calculate the metric once time for one batch.
Good suggestion. I think calculating for every single message is better, the
metrics `publish latency` will be more precise when users set
`batchingMaxPublishDelay` to a large value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[PR] [improve][admin] Add authorization test for schema and align auth for transaction [pulsar]
liangyepianzhou opened a new pull request, #22399: URL: https://github.com/apache/pulsar/pull/22399 ### Motivation 1. Add authorization tests for schema and transaction. 2. Aligin authorization for transaction. ### Modifications 1. Add authorization tests for schema and transaction. 2. Aligin authorization for transaction. 1. Check the role permission of the topic when getting stats for TB or TC. 2. Does not check the permission when fetching partition metadata to get stats for TC. (Draft, the another approach is to make the users grant the role lookup permission for the TC. It is a cognitive cost for users to learn about what is TC.) ### Verifying this change - [ ] Make sure that the change passes the CI checks. *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Does this pull request potentially affect one of the following parts: *If the box was checked, please highlight the changes* - [ ] Dependencies (add or upgrade a dependency) - [ ] The public API - [ ] The schema - [ ] The default values of configurations - [ ] The threading model - [ ] The binary protocol - [ ] The REST endpoints - [ ] The admin CLI options - [ ] The metrics - [ ] Anything that affects deployment ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` ### Matching PR in forked repository PR in forked repository: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][client] client side memory leak when call ProducerBase.sendAsync(Message) [pulsar]
poorbarcode commented on code in PR #22393:
URL: https://github.com/apache/pulsar/pull/22393#discussion_r1548004698
##
pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java:
##
@@ -418,16 +419,14 @@ public void sendComplete(Exception e) {
stats.incrementNumAcksReceived(latencyNanos);
}
} finally {
-interceptorMessage.getDataBuffer().release();
+ReferenceCountUtil.safeRelease(payloadInCurrentMsg);
Review Comment:
And fixed the inaccurate metrics: latencyNanos
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [fix][client] client side memory leak when call ProducerBase.sendAsync(Message) [pulsar]
poorbarcode commented on code in PR #22393:
URL: https://github.com/apache/pulsar/pull/22393#discussion_r1547984654
##
pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java:
##
@@ -418,16 +419,14 @@ public void sendComplete(Exception e) {
stats.incrementNumAcksReceived(latencyNanos);
}
} finally {
-interceptorMessage.getDataBuffer().release();
+ReferenceCountUtil.safeRelease(payloadInCurrentMsg);
Review Comment:
Good suggestion, improved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Add best consume practice [pulsar-site]
visortelle commented on code in PR #875: URL: https://github.com/apache/pulsar-site/pull/875#discussion_r1547954424 ## static/img/blog-consume-best-practice/Ack-hole.png: ## Review Comment: +1 Mermaid is suitable for most cases, but for some complex diagrams (if any), it also could be [draw.io](https://github.com/jgraph/drawio) (diagrams.net) that could be stored in the repository as .xml files. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Pulsar Manager v0.4.0 in kubernetes still can not persist any data [pulsar-helm-chart]
lhotari commented on issue #470: URL: https://github.com/apache/pulsar-helm-chart/issues/470#issuecomment-2032102604 Contributions are welcome to fix this issue! /cc @Mortom123 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Argo CD unable to generate template with pulsar chart 3.3.0 [pulsar-helm-chart]
lhotari commented on issue #471: URL: https://github.com/apache/pulsar-helm-chart/issues/471#issuecomment-2032100762 @yonatanamz the fixes will be included in the upcoming 3.4.0 version of the Apache Pulsar Helm chart. This is currently in release candidate voting stage. Please help validate the Apache Pulsar Helm chart release 3.4.0: https://lists.apache.org/thread/t4vlvk0nwd3gzo50t466cqy5h95l81gb . The release vote email contains instructions how to use the release candidate version. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add best consume practice [pulsar-site]
asafm commented on code in PR #875: URL: https://github.com/apache/pulsar-site/pull/875#discussion_r1547583880 ## docs/tutorials-redeliver-messages.md: ## @@ -0,0 +1,236 @@ +--- +Id: tutorials-redeliver-messages +title: Consume best practice +sidebar_label: "Consume best practice" +description: Learn how to consume messages and redeliver unacknowledged messages in Pulsar. +--- + +# Consume Best Practice + +## Background Knowledge + +### Subscription Types + +Pulsar is a distributed message system where messages can be sent to topics by producers and consumed by consumers. +Consumers can subscribe to the topics in four ways (subscription types): + +* **Exclusive** +* **Failover** +* **Shared** +* **Key-shared** + +The messages are consumed in order for a single partition in Exclusive and Failover modes and out of order for Shared and +Key-shared mode. The main difference between Exclusive and Failover is that in Exclusive mode, the consumer is exclusive +to the entire topic, while in Failover mode, the consumer is exclusive to only one partition. Failover mode allows backup Review Comment: Even if the topic has partitions across brokers? How? ## docs/tutorials-redeliver-messages.md: ## @@ -0,0 +1,236 @@ +--- +Id: tutorials-redeliver-messages +title: Consume best practice +sidebar_label: "Consume best practice" +description: Learn how to consume messages and redeliver unacknowledged messages in Pulsar. +--- + +# Consume Best Practice + +## Background Knowledge + +### Subscription Types + +Pulsar is a distributed message system where messages can be sent to topics by producers and consumed by consumers. +Consumers can subscribe to the topics in four ways (subscription types): + +* **Exclusive** +* **Failover** +* **Shared** +* **Key-shared** + +The messages are consumed in order for a single partition in Exclusive and Failover modes and out of order for Shared and +Key-shared mode. The main difference between Exclusive and Failover is that in Exclusive mode, the consumer is exclusive +to the entire topic, while in Failover mode, the consumer is exclusive to only one partition. Failover mode allows backup Review Comment: Back--> Stand-by. The other consumers which are not active, are also connected to the partition in stand-by. ## docs/tutorials-redeliver-messages.md: ## @@ -0,0 +1,236 @@ +--- +Id: tutorials-redeliver-messages +title: Consume best practice +sidebar_label: "Consume best practice" +description: Learn how to consume messages and redeliver unacknowledged messages in Pulsar. +--- + +# Consume Best Practice + +## Background Knowledge + +### Subscription Types + +Pulsar is a distributed message system where messages can be sent to topics by producers and consumed by consumers. +Consumers can subscribe to the topics in four ways (subscription types): + +* **Exclusive** +* **Failover** +* **Shared** +* **Key-shared** + +The messages are consumed in order for a single partition in Exclusive and Failover modes and out of order for Shared and +Key-shared mode. The main difference between Exclusive and Failover is that in Exclusive mode, the consumer is exclusive +to the entire topic, while in Failover mode, the consumer is exclusive to only one partition. Failover mode allows backup +consumer connections that are not consumed. The main difference between Shared and Key-shared is whether their dispatch +strategy is implemented via a key. For more information about subscription type, refer to the [Pulsar website](https://pulsar.apache.org/docs/3.2.x/concepts-messaging/). + + +### Acknowledgment + +The messages should be acknowledged after they are fully consumed and processed, and then the messages would not be received +for the same subscription again. Pulsar provides two ways to acknowledge messages: + +* **Cumulative acknowledgment** +* **Individual acknowledgment** + +Cumulative acknowledgment receives a message or a message id as a parameter and marks the messages before the message as +consumed for this subscription. For multiple-partition topics, the cumulative acknowledgment will work for the single +partition without impacting other partitions. Individual acknowledgment receives a message or a message id as a parameter +and only marks this message as consumed for this subscription. + + +### Messages Redeliver Mechanism + +There might be instances where the received messages cannot be processed at this time or some errors happened during processing. +The client needs to redeliver the unacknowledged messages or a particular message after a delay or immediately. +Pulsar provides at-least-once semantics when the client does not enable transaction because the client may cache some Review Comment: I provided a better explanation above. What is the
Re: [PR] pulsar-manager adding support for existing secret [pulsar-helm-chart]
lhotari commented on PR #478: URL: https://github.com/apache/pulsar-helm-chart/pull/478#issuecomment-2032075038 @robertdumitrescu The next release is in voting stage. Please help validate the release: https://lists.apache.org/thread/t4vlvk0nwd3gzo50t466cqy5h95l81gb . The release voting email shows how you can use the release candidate version. Would you be able to test that existing secret works as expected? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar-helm-chart) annotated tag pulsar-3.4.0-candidate-1 updated (d9e6583 -> cebbc1f)
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a change to annotated tag pulsar-3.4.0-candidate-1 in repository https://gitbox.apache.org/repos/asf/pulsar-helm-chart.git *** WARNING: tag pulsar-3.4.0-candidate-1 was modified! *** from d9e6583 (commit) to cebbc1f (tag) tagging d9e65836e8374f472ae85b79b318b8c1ff17352d (commit) replaces pulsar-3.3.1 by Lari Hotari on Tue Apr 2 16:31:22 2024 +0300 - Log - Apache Pulsar Helm Chart 3.4.0-candidate-1 -BEGIN PGP SIGNATURE- iQIzBAABCgAdFiEExmPS9k2iygnbKNmr0/pn1SLFUlYFAmYMCKoACgkQ0/pn1SLF UlauWxAA6twp3rh6PRYqnZxszv0ByTt+7CSvGZ10IEH/DHcvg1+uZI/69bCil9vL 72xhEB4MQl+GK9j67Uhcx51ligg8l24++3LgYax4R1rT2bASpE0BF+OCgC1W2VW4 V4Dhz1glinrBD/FuvMUJ01Csp961CuJlham4D9x88fod2nOFfzslOh9qDCjv7mV2 5aDrGA7rXTuRe6a+2ljyFgWlCQhqAwXfJAks3sU0x/FSCO1HxAwZlLLqIfEy5JC7 sTcUm1nhGovda+mzaKv2S6omSYY0csdGXhiwPa0nb7Gep6JWIeGrODEbJ8qYM7G4 9A4xXWr/VhOSqEXRaTgOlIw9UHaeNpq3gfq4GPp4qioLiuzFhr7gX4tpghCKHiwJ R74GpTLMAMYfbYGH5ov8f+hGDJUTnloTFkeglfC1e8i7TrCxy94WXQBeJoMbsDyJ mNW5qMbjr+X+XXQ+fYomTyHXny9CjfpRmGZWihYwLtpm8CxiBxImWksjgFuoL7vC aai0gmapKOfZ3oJ8s3cjXBy/iyfodNnEKyuMRm0V9+VV5+wZ/voglVGwgp+8/VH8 c0qfTve+VSUgYauqX5kzhItpawNSYFR7KrHN2JCyKpKatHg4OdpY1UnF1EbVfjAM 1hDDLICobjGOXamROgAdO2jCiPiFsvMlwIW1DKTLVYNFtNva4uE= =PbCE -END PGP SIGNATURE- --- No new revisions were added by this update. Summary of changes:
svn commit: r68260 - /dev/pulsar/helm-chart/3.4.0-candidate-1/
Author: lhotari Date: Tue Apr 2 13:32:11 2024 New Revision: 68260 Log: Add artifacts for Helm Chart 3.4.0-candidate-1 Added: dev/pulsar/helm-chart/3.4.0-candidate-1/index.yaml dev/pulsar/helm-chart/3.4.0-candidate-1/pulsar-3.4.0.tgz (with props) dev/pulsar/helm-chart/3.4.0-candidate-1/pulsar-3.4.0.tgz.asc dev/pulsar/helm-chart/3.4.0-candidate-1/pulsar-3.4.0.tgz.prov dev/pulsar/helm-chart/3.4.0-candidate-1/pulsar-3.4.0.tgz.sha512 dev/pulsar/helm-chart/3.4.0-candidate-1/pulsar-chart-3.4.0-source.tar.gz (with props) dev/pulsar/helm-chart/3.4.0-candidate-1/pulsar-chart-3.4.0-source.tar.gz.asc dev/pulsar/helm-chart/3.4.0-candidate-1/pulsar-chart-3.4.0-source.tar.gz.sha512 Added: dev/pulsar/helm-chart/3.4.0-candidate-1/index.yaml == --- dev/pulsar/helm-chart/3.4.0-candidate-1/index.yaml (added) +++ dev/pulsar/helm-chart/3.4.0-candidate-1/index.yaml Tue Apr 2 13:32:11 2024 @@ -0,0 +1,555 @@ +apiVersion: v1 +entries: + pulsar: + - apiVersion: v2 +appVersion: 3.0.4 +created: "2024-04-02T16:32:03.130116+03:00" +dependencies: +- condition: kube-prometheus-stack.enabled + name: kube-prometheus-stack + repository: https://prometheus-community.github.io/helm-charts + version: 56.x.x +description: Apache Pulsar Helm chart for Kubernetes +digest: c61ddbd5c9673b8212aa64e466196d0372a2cb287f77540b6484ed5417513874 +home: https://pulsar.apache.org +icon: https://pulsar.apache.org/img/pulsar.svg +kubeVersion: '>=1.21.0-0' +maintainers: +- email: d...@pulsar.apache.org + name: The Apache Pulsar Team +name: pulsar +sources: +- https://github.com/apache/pulsar +- https://github.com/apache/pulsar-helm-chart +urls: +- https://dist.apache.org/repos/dist/dev/pulsar/helm-chart/3.4.0-candidate-1/pulsar-3.4.0.tgz +version: 3.4.0 + - apiVersion: v2 +appVersion: 3.0.3 +created: "2024-03-21T11:30:52.258959+02:00" +dependencies: +- condition: kube-prometheus-stack.enabled + name: kube-prometheus-stack + repository: https://prometheus-community.github.io/helm-charts + version: 56.x.x +description: Apache Pulsar Helm chart for Kubernetes +digest: 01e1822e10811352b2ed25e0b2d6a9d9a90af8efc5d32c82feb9301fab21fc83 +home: https://pulsar.apache.org +icon: https://pulsar.apache.org/img/pulsar.svg +kubeVersion: '>=1.21.0-0' +maintainers: +- email: d...@pulsar.apache.org + name: The Apache Pulsar Team +name: pulsar +sources: +- https://github.com/apache/pulsar +- https://github.com/apache/pulsar-helm-chart +urls: +- https://archive.apache.org/dist/pulsar/helm-chart/3.3.1/pulsar-3.3.1.tgz +version: 3.3.1 + - apiVersion: v2 +appVersion: 3.0.2 +created: "2024-02-27T16:00:17.329495+02:00" +dependencies: +- condition: kube-prometheus-stack.enabled + name: kube-prometheus-stack + repository: https://prometheus-community.github.io/helm-charts + version: 56.x.x +description: Apache Pulsar Helm chart for Kubernetes +digest: 2c0bcd415403d55f2cd941d543b9801e6daed8b23d7dd4f18bce5a6a89e2 +home: https://pulsar.apache.org +icon: https://pulsar.apache.org/img/pulsar.svg +kubeVersion: '>=1.21.0-0' +maintainers: +- email: d...@pulsar.apache.org + name: The Apache Pulsar Team +name: pulsar +sources: +- https://github.com/apache/pulsar +- https://github.com/apache/pulsar-helm-chart +urls: +- https://archive.apache.org/dist/pulsar/helm-chart/3.3.0/pulsar-3.3.0.tgz +version: 3.3.0 + - apiVersion: v2 +appVersion: 3.0.2 +created: "2024-01-25T17:32:19.5781+02:00" +dependencies: +- condition: kube-prometheus-stack.enabled + name: kube-prometheus-stack + repository: https://prometheus-community.github.io/helm-charts + version: 41.x.x +description: Apache Pulsar Helm chart for Kubernetes +digest: 4ba015f2dd3cfdc805179af88284d22b4723e32a82fd1f7c288db4fcca7d18f0 +home: https://pulsar.apache.org +icon: https://pulsar.apache.org/img/pulsar.svg +maintainers: +- email: d...@pulsar.apache.org + name: The Apache Pulsar Team +name: pulsar +sources: +- https://github.com/apache/pulsar +- https://github.com/apache/pulsar-helm-chart +urls: +- https://archive.apache.org/dist/pulsar/helm-chart/3.2.0/pulsar-3.2.0.tgz +version: 3.2.0 + - apiVersion: v2 +appVersion: 3.0.2 +created: "2023-12-21T10:37:11.997345+02:00" +dependencies: +- condition: kube-prometheus-stack.enabled + name: kube-prometheus-stack + repository: https://prometheus-community.github.io/helm-charts + version: 41.x.x +description: Apache Pulsar Helm chart for Kubernetes +digest: ee6f0e7ef3ceaf5957144ac77a2dcfa207a0819d5d1cd2123d6b04e65c3b4021 +home: https://pulsar.apache.org +icon:
svn commit: r68259 - /dev/pulsar/helm-chart/3.4.0-candidate-1/
Author: lhotari Date: Tue Apr 2 13:32:01 2024 New Revision: 68259 Log: Add directory for pulsar-helm-chart 3.4.0-candidate-1 release Added: dev/pulsar/helm-chart/3.4.0-candidate-1/
(pulsar-helm-chart) branch master updated: Chart: Bump version to 3.4.0
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar-helm-chart.git The following commit(s) were added to refs/heads/master by this push: new d9e6583 Chart: Bump version to 3.4.0 d9e6583 is described below commit d9e65836e8374f472ae85b79b318b8c1ff17352d Author: Lari Hotari AuthorDate: Tue Apr 2 16:31:14 2024 +0300 Chart: Bump version to 3.4.0 --- charts/pulsar/Chart.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/charts/pulsar/Chart.yaml b/charts/pulsar/Chart.yaml index b2c399b..941483a 100644 --- a/charts/pulsar/Chart.yaml +++ b/charts/pulsar/Chart.yaml @@ -21,7 +21,7 @@ apiVersion: v2 appVersion: "3.0.4" description: Apache Pulsar Helm chart for Kubernetes name: pulsar -version: 3.3.1 +version: 3.4.0 kubeVersion: ">=1.21.0-0" home: https://pulsar.apache.org sources:
Re: [PR] Upgrade appVersion to 3.0.4 to use Pulsar 3.0.4 by default [pulsar-helm-chart]
lhotari merged PR #484: URL: https://github.com/apache/pulsar-helm-chart/pull/484 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar-helm-chart) branch master updated: Upgrade appVersion to 3.0.4 to use Pulsar 3.0.4 by default (#484)
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar-helm-chart.git The following commit(s) were added to refs/heads/master by this push: new a8776fd Upgrade appVersion to 3.0.4 to use Pulsar 3.0.4 by default (#484) a8776fd is described below commit a8776fd76cdac46aa5962b7281fbd133d36b80fd Author: Lari Hotari AuthorDate: Tue Apr 2 06:28:38 2024 -0700 Upgrade appVersion to 3.0.4 to use Pulsar 3.0.4 by default (#484) --- charts/pulsar/Chart.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/charts/pulsar/Chart.yaml b/charts/pulsar/Chart.yaml index 8272e57..b2c399b 100644 --- a/charts/pulsar/Chart.yaml +++ b/charts/pulsar/Chart.yaml @@ -18,7 +18,7 @@ # apiVersion: v2 -appVersion: "3.0.3" +appVersion: "3.0.4" description: Apache Pulsar Helm chart for Kubernetes name: pulsar version: 3.3.1
Re: [PR] [fix][client] client side memory leak when call ProducerBase.sendAsync(Message) [pulsar]
gaoran10 commented on code in PR #22393:
URL: https://github.com/apache/pulsar/pull/22393#discussion_r1547838994
##
pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java:
##
@@ -418,16 +419,14 @@ public void sendComplete(Exception e) {
stats.incrementNumAcksReceived(latencyNanos);
}
} finally {
-interceptorMessage.getDataBuffer().release();
+ReferenceCountUtil.safeRelease(payloadInCurrentMsg);
Review Comment:
Maybe we can calculate the metric once time for one batch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [fix][client] client side memory leak when call ProducerBase.sendAsync(Message) [pulsar]
gaoran10 commented on code in PR #22393:
URL: https://github.com/apache/pulsar/pull/22393#discussion_r1547838994
##
pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java:
##
@@ -418,16 +419,14 @@ public void sendComplete(Exception e) {
stats.incrementNumAcksReceived(latencyNanos);
}
} finally {
-interceptorMessage.getDataBuffer().release();
+ReferenceCountUtil.safeRelease(payloadInCurrentMsg);
Review Comment:
Maybe we can calculate the metric once time for one batch message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Add support for HasMessageAvailable [pulsar-dotpulsar]
blankensteiner commented on PR #207: URL: https://github.com/apache/pulsar-dotpulsar/pull/207#issuecomment-2031919693 Hi @smbecker, does this extension method float your boat? :-) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Multi threaded message consumer issue [pulsar-dotpulsar]
entvex commented on issue #211: URL: https://github.com/apache/pulsar-dotpulsar/issues/211#issuecomment-2031909016 Dear @savan-rangrej. Thanks for the bug report! When you run into this issue, are you using a partitioned topic? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [D] async/sync mode in pulsar producer and consumer [pulsar]
GitHub user lhotari edited a comment on the discussion: async/sync mode in pulsar producer and consumer > 1. For synchronous mode we use send() method > 2. For asynchronous mode we use sendAsync() method @ragaur-tibco yes. > if answer to the above two points is yes then why is send() method internally > calling the sendAsync() method @ragaur-tibco I don't really understand the point of the question. Do you have a problem with this implementation? What are your expectations for send and sendAsync? Is there some expectation that isn't met? With the current implementation in the Pulsar Java client, it simply fulfills the requirements when send() internally calls sendAsync() method. Perhaps you have different expectations or requirements. GitHub link: https://github.com/apache/pulsar/discussions/22382#discussioncomment-8982733 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
(pulsar-site) branch main updated: Shuffle testimonials before displaying them (#876)
This is an automated email from the ASF dual-hosted git repository.
visortelle pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/pulsar-site.git
The following commit(s) were added to refs/heads/main by this push:
new 794145f3c66a Shuffle testimonials before displaying them (#876)
794145f3c66a is described below
commit 794145f3c66a2d4baa083323ffb13bed128ab54c
Author: Lari Hotari
AuthorDate: Tue Apr 2 04:52:11 2024 -0700
Shuffle testimonials before displaying them (#876)
* Shuffle testimonials before displaying them
* Address review comment
---
src/components/pages/HomePage/Users/Users.tsx | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/src/components/pages/HomePage/Users/Users.tsx
b/src/components/pages/HomePage/Users/Users.tsx
index 54dfc91528a1..4d63df2a787c 100644
--- a/src/components/pages/HomePage/Users/Users.tsx
+++ b/src/components/pages/HomePage/Users/Users.tsx
@@ -1,4 +1,5 @@
import React from 'react';
+import _ from 'lodash'
import Button from '@site/src/components/ui/Button/Button';
import Slider from '@site/src/components/ui/Slider/Slider';
@@ -12,6 +13,7 @@ import BrowserOnly from '@docusaurus/BrowserOnly';
const Users: React.FC = () => {
const { siteConfig } = useDocusaurusContext();
+ const shuffledTestimonials = React.useMemo(() => _.shuffle(testimonials),
[testimonials]);
return (
@@ -37,7 +39,7 @@ const Users: React.FC = () => {
{() => (
1000} slidesToShow={2}>
- {Object.values(testimonials).flat().map((caseStudy, i) => (
+ {Object.values(shuffledTestimonials).flat().map((caseStudy, i)
=> (
Re: [PR] Shuffle testimonials before displaying them [pulsar-site]
visortelle merged PR #876: URL: https://github.com/apache/pulsar-site/pull/876 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][client] client side memory leak when call ProducerBase.sendAsync(Message) [pulsar]
codelipenghui commented on code in PR #22393:
URL: https://github.com/apache/pulsar/pull/22393#discussion_r1547722641
##
pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java:
##
@@ -418,16 +419,14 @@ public void sendComplete(Exception e) {
stats.incrementNumAcksReceived(latencyNanos);
}
} finally {
-interceptorMessage.getDataBuffer().release();
+ReferenceCountUtil.safeRelease(payloadInCurrentMsg);
Review Comment:
I tried a patch on my laptop. It looks like we also missed the stats and
metrics update in handing the batch messages.
```diff
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleProducerConsumerTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleProducerConsumerTest.java
index 4c106d39e7..c872a4c79c 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleProducerConsumerTest.java
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleProducerConsumerTest.java
@@ -69,6 +69,7 @@ import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicLong;
import java.util.concurrent.atomic.AtomicReference;
@@ -92,18 +93,10 @@ import org.apache.pulsar.broker.service.ServerCnx;
import org.apache.pulsar.broker.service.persistent.PersistentTopic;
import org.apache.pulsar.client.admin.PulsarAdminException;
import org.apache.pulsar.client.api.schema.GenericRecord;
-import org.apache.pulsar.client.impl.ClientBuilderImpl;
-import org.apache.pulsar.client.impl.ConsumerBase;
-import org.apache.pulsar.client.impl.ConsumerImpl;
-import org.apache.pulsar.client.impl.MessageIdImpl;
-import org.apache.pulsar.client.impl.MessageImpl;
-import org.apache.pulsar.client.impl.MultiTopicsConsumerImpl;
-import org.apache.pulsar.client.impl.PartitionedProducerImpl;
-import org.apache.pulsar.client.impl.ProducerImpl;
-import org.apache.pulsar.client.impl.TopicMessageImpl;
-import org.apache.pulsar.client.impl.TypedMessageBuilderImpl;
+import org.apache.pulsar.client.impl.*;
import org.apache.pulsar.client.impl.crypto.MessageCryptoBc;
import org.apache.pulsar.client.impl.schema.writer.AvroWriter;
+import org.apache.pulsar.common.allocator.PulsarByteBufAllocator;
import org.apache.pulsar.common.api.EncryptionContext;
import org.apache.pulsar.common.api.EncryptionContext.EncryptionKey;
import org.apache.pulsar.common.api.proto.MessageMetadata;
@@ -4692,4 +4685,81 @@ public class SimpleProducerConsumerTest extends
ProducerConsumerBase {
consumer.close();
admin.topics().delete(topic, false);
}
+
+@DataProvider(name = "enableBatchSend")
+public Object[][] enableBatchSend() {
+return new Object[][]{
+{true},
+{false}
+};
+}
+
+@Test(dataProvider = "enableBatchSend")
+public void testPublishWithCreateMessageManually(boolean
enableBatchSend) throws Exception {
+// Create an interceptor to verify the ref count of Message.payload
is as expected.
+AtomicBoolean payloadWasReleasedWhenIntercept = new
AtomicBoolean(false);
+ProducerInterceptor interceptor = new ProducerInterceptor(){
+
+@Override
+public void close() {
+
+}
+@Override
+public Message beforeSend(Producer producer, Message message) {
+MessageImpl msgImpl = (MessageImpl) message;
+log.info("payload.refCnf before send: {}",
msgImpl.getDataBuffer().refCnt());
+if (msgImpl.getDataBuffer().refCnt() < 1) {
+payloadWasReleasedWhenIntercept.set(true);
+}
+return message;
+}
+
+@Override
+public void onSendAcknowledgement(Producer producer, Message
message, MessageId msgId,
+ Throwable exception) {
+MessageImpl msgImpl = (MessageImpl) message;
+log.info("payload.refCnf on send acknowledgement: {}",
msgImpl.getDataBuffer().refCnt());
+if (msgImpl.getDataBuffer().refCnt() < 1) {
+payloadWasReleasedWhenIntercept.set(true);
+}
+}
+};
+
+final String topic =
BrokerTestUtil.newUniqueName("persistent://my-property/my-ns/tp");
+admin.topics().createNonPartitionedTopic(topic);
+ProducerBase
Re: [PR] [improve] [broker] Do not print an Error log when responding to `HTTP-404` when calling `Admin API` and the topic does not exist. [pulsar]
jaak-pruulmann-sympower commented on code in PR #21995:
URL: https://github.com/apache/pulsar/pull/21995#discussion_r1547715095
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/NonPersistentTopics.java:
##
@@ -132,7 +132,7 @@ public void getInternalStats(
})
.thenAccept(asyncResponse::resume)
.exceptionally(ex -> {
-if (!isRedirectException(ex)) {
+if (!isNot307And404Exception(ex)) {
Review Comment:
Looks like this line and all the others where `!isRedirectException(ex)` has
been replaced with `!isNot307..` are wrong, as the new line should NOT have the
negation exclamation mark any more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Shuffle testimonials before displaying them [pulsar-site]
lhotari commented on PR #876: URL: https://github.com/apache/pulsar-site/pull/876#issuecomment-2031785902 thanks @visortelle , I applied the suggestion. PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar-site) branch main updated: Recommend ZK for Pulsar standalone (#871)
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/pulsar-site.git The following commit(s) were added to refs/heads/main by this push: new 19a98625af5e Recommend ZK for Pulsar standalone (#871) 19a98625af5e is described below commit 19a98625af5ef0bafaffe6f8ea60a9a9c7417819 Author: Kiryl Valkovich AuthorDate: Tue Apr 2 15:34:48 2024 +0400 Recommend ZK for Pulsar standalone (#871) --- docs/getting-started-docker-compose.md | 1 + docs/getting-started-docker.md | 16 versioned_docs/version-3.0.x/getting-started-docker.md | 16 .../version-3.2.x/getting-started-docker-compose.md | 1 + versioned_docs/version-3.2.x/getting-started-docker.md | 16 5 files changed, 50 insertions(+) diff --git a/docs/getting-started-docker-compose.md b/docs/getting-started-docker-compose.md index 3f06c5d844fe..b3048013c5dd 100644 --- a/docs/getting-started-docker-compose.md +++ b/docs/getting-started-docker-compose.md @@ -114,6 +114,7 @@ services: ## Step 2: Create a Pulsar cluster As preparation, create data directories and change the data directory ownership to uid(1) which is the default user id used in the Pulsar Docker container. + ```bash sudo mkdir -p ./data/zookeeper ./data/bookkeeper # this step might not be necessary on other than Linux platforms diff --git a/docs/getting-started-docker.md b/docs/getting-started-docker.md index 785fb8f9037e..302ebbf007eb 100644 --- a/docs/getting-started-docker.md +++ b/docs/getting-started-docker.md @@ -56,6 +56,22 @@ bin/pulsar standalone +:::tip + +You may encounter issues with the default RocksDB metadata store. + +We recommend you consider using the following environment variable to use ZooKeeper as the metadata store: + +``` +... +-e PULSAR_STANDALONE_USE_ZOOKEEPER=1 \ +... +``` + +Don't apply this fix for existing Pulsar standalone instances if you don't want to loose your data. + +::: + If you want to change Pulsar configurations and start Pulsar, run the following command by passing environment variables with the `PULSAR_PREFIX_` prefix. See [default configuration file](https://github.com/apache/pulsar/blob/e6b12c64b043903eb5ff2dc5186fe8030f157cfc/conf/standalone.conf) for more details. mdx-code-block diff --git a/versioned_docs/version-3.0.x/getting-started-docker.md b/versioned_docs/version-3.0.x/getting-started-docker.md index e8d706f0f7ca..361746f08988 100644 --- a/versioned_docs/version-3.0.x/getting-started-docker.md +++ b/versioned_docs/version-3.0.x/getting-started-docker.md @@ -16,6 +16,22 @@ For macOS, Linux, and Windows, run the following command to start Pulsar within docker run -it -p 6650:6650 -p 8080:8080 --mount source=pulsardata,target=/pulsar/data --mount source=pulsarconf,target=/pulsar/conf apachepulsar/pulsar:@pulsar:version@ bin/pulsar standalone ``` +:::tip + +You may encounter issues with the default RocksDB metadata store. + +We recommend you consider using the following environment variable to use ZooKeeper as the metadata store: + +``` +... +-e PULSAR_STANDALONE_USE_ZOOKEEPER=1 \ +... +``` + +Don't apply this fix for existing Pulsar standalone instances if you don't want to loose your data. + +::: + If you want to change Pulsar configurations and start Pulsar, run the following command by passing environment variables with the `PULSAR_PREFIX_` prefix. See [default configuration file](https://github.com/apache/pulsar/blob/e6b12c64b043903eb5ff2dc5186fe8030f157cfc/conf/standalone.conf) for more details. ```shell diff --git a/versioned_docs/version-3.2.x/getting-started-docker-compose.md b/versioned_docs/version-3.2.x/getting-started-docker-compose.md index 3f06c5d844fe..b3048013c5dd 100644 --- a/versioned_docs/version-3.2.x/getting-started-docker-compose.md +++ b/versioned_docs/version-3.2.x/getting-started-docker-compose.md @@ -114,6 +114,7 @@ services: ## Step 2: Create a Pulsar cluster As preparation, create data directories and change the data directory ownership to uid(1) which is the default user id used in the Pulsar Docker container. + ```bash sudo mkdir -p ./data/zookeeper ./data/bookkeeper # this step might not be necessary on other than Linux platforms diff --git a/versioned_docs/version-3.2.x/getting-started-docker.md b/versioned_docs/version-3.2.x/getting-started-docker.md index 785fb8f9037e..302ebbf007eb 100644 --- a/versioned_docs/version-3.2.x/getting-started-docker.md +++ b/versioned_docs/version-3.2.x/getting-started-docker.md @@ -56,6 +56,22 @@ bin/pulsar standalone +:::tip + +You may encounter issues with the default RocksDB metadata store. + +We recommend you consider using the following environment variable to use ZooKeeper as the metadata store: + +``` +... +-e PULSAR_STANDALONE_USE_ZOOKEEPER=1 \ +... +``` + +Don't apply this
Re: [PR] Recommend ZK for Pulsar standalone [pulsar-site]
lhotari merged PR #871: URL: https://github.com/apache/pulsar-site/pull/871 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Recommend ZK for Pulsar standalone [pulsar-site]
lhotari commented on code in PR #871: URL: https://github.com/apache/pulsar-site/pull/871#discussion_r1547689275 ## versioned_docs/version-3.2.x/getting-started-docker.md: ## @@ -56,6 +56,22 @@ bin/pulsar standalone +:::tip + +You may encounter issues with the default RocksDB metadata store after restarting Pulsar standalone. + +We recommend you consider using the following environment variable to use ZooKeeper as the metadata store: Review Comment: I'll merge this PR with the current content. We can come back to this later once there's a better resolution. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Recommend ZK for Pulsar standalone [pulsar-site]
lhotari commented on code in PR #871: URL: https://github.com/apache/pulsar-site/pull/871#discussion_r1547687082 ## versioned_docs/version-3.2.x/getting-started-docker.md: ## @@ -56,6 +56,22 @@ bin/pulsar standalone +:::tip + +You may encounter issues with the default RocksDB metadata store after restarting Pulsar standalone. + +We recommend you consider using the following environment variable to use ZooKeeper as the metadata store: Review Comment: > I don't think it's a good idea to recommend not to use a default setting. If we have a bug in default configuration, we must fix it. If it cannot be fixed, we will have to change the defaults. @merlimat I agree. Since we cannot change already released versions and users are hitting problems, I think it's worth documenting the workaround until there's a better solution. It seems that there isn't a nice way to solve this problem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [improve][test] Add operation authentication test for namespace API [pulsar]
coderzc opened a new pull request, #22398: URL: https://github.com/apache/pulsar/pull/22398 ### Motivation Add operation authentication test for namespace API. ### Modifications Add operation authentication test. ### Verifying this change - [ ] Make sure that the change passes the CI checks. *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Does this pull request potentially affect one of the following parts: *If the box was checked, please highlight the changes* - [ ] Dependencies (add or upgrade a dependency) - [ ] The public API - [ ] The schema - [ ] The default values of configurations - [ ] The threading model - [ ] The binary protocol - [ ] The REST endpoints - [ ] The admin CLI options - [ ] The metrics - [ ] Anything that affects deployment ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` ### Matching PR in forked repository PR in forked repository: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [D] async/sync mode in pulsar producer and consumer [pulsar]
GitHub user lhotari edited a comment on the discussion: async/sync mode in pulsar producer and consumer > 1. For synchronous mode we use send() method > 2. For asynchronous mode we use sendAsync() method @ragaur-tibco yes. > if answer to the above two points is yes then why is send() method internally > calling the sendAsync() method @ragaur-tibco I don't really understand the point of the question. Do you have a problem with this implementation? What are your expectations for send and sendAsync? Is there some expectation that isn't met? With the current mode in the Pulsar Java client, it simply fulfills the requirements when send() internally calls sendAsync() method. Perhaps you have different expectations or requirements. GitHub link: https://github.com/apache/pulsar/discussions/22382#discussioncomment-8982733 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
