Re: [PR] [fix][cli] Fix the pulsar-daemon parameter passthrough syntax [pulsar]
coderzc commented on code in PR #22905: URL: https://github.com/apache/pulsar/pull/22905#discussion_r1639290804 ## bin/pulsar-daemon: ## @@ -157,7 +157,7 @@ start () echo starting $command, logging to $logfile echo Note: Set immediateFlush to true in conf/log4j2.yaml will guarantee the logging event is flushing to disk immediately. The default behavior is switched off due to performance considerations. pulsar=$PULSAR_HOME/bin/pulsar -nohup $pulsar $command "$1" > "$out" 2>&1 < /dev/null & +nohup $pulsar $command $1 > "$out" 2>&1 < /dev/null & Review Comment: @lhotari In my local machine, it cannot be started using quotation marks. I need to remove the quotation to make it work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][cli] Fix the pulsar-daemon parameter passthrough syntax [pulsar]
coderzc commented on code in PR #22905: URL: https://github.com/apache/pulsar/pull/22905#discussion_r1639290804 ## bin/pulsar-daemon: ## @@ -157,7 +157,7 @@ start () echo starting $command, logging to $logfile echo Note: Set immediateFlush to true in conf/log4j2.yaml will guarantee the logging event is flushing to disk immediately. The default behavior is switched off due to performance considerations. pulsar=$PULSAR_HOME/bin/pulsar -nohup $pulsar $command "$1" > "$out" 2>&1 < /dev/null & +nohup $pulsar $command $1 > "$out" 2>&1 < /dev/null & Review Comment: In my local machine, it cannot be started using quotation marks. I need to remove the quotation to make it work. ## bin/pulsar-daemon: ## @@ -157,7 +157,7 @@ start () echo starting $command, logging to $logfile echo Note: Set immediateFlush to true in conf/log4j2.yaml will guarantee the logging event is flushing to disk immediately. The default behavior is switched off due to performance considerations. pulsar=$PULSAR_HOME/bin/pulsar -nohup $pulsar $command "$1" > "$out" 2>&1 < /dev/null & +nohup $pulsar $command $1 > "$out" 2>&1 < /dev/null & Review Comment: @lhotari In my local machine, it cannot be started using quotation marks. I need to remove the quotation to make it work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][cli] Fix the pulsar-daemon parameter passthrough syntax [pulsar]
lhotari commented on code in PR #22905: URL: https://github.com/apache/pulsar/pull/22905#discussion_r1639286485 ## bin/pulsar-daemon: ## @@ -157,7 +157,7 @@ start () echo starting $command, logging to $logfile echo Note: Set immediateFlush to true in conf/log4j2.yaml will guarantee the logging event is flushing to disk immediately. The default behavior is switched off due to performance considerations. pulsar=$PULSAR_HOME/bin/pulsar -nohup $pulsar $command "$1" > "$out" 2>&1 < /dev/null & +nohup $pulsar $command $1 > "$out" 2>&1 < /dev/null & Review Comment: the solution for restart needs also fixing. I can push changes directly to this PR to help in fixing that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][cli] Fix the pulsar-daemon parameter passthrough syntax [pulsar]
lhotari commented on code in PR #22905: URL: https://github.com/apache/pulsar/pull/22905#discussion_r1639282466 ## bin/pulsar-daemon: ## @@ -157,7 +157,7 @@ start () echo starting $command, logging to $logfile echo Note: Set immediateFlush to true in conf/log4j2.yaml will guarantee the logging event is flushing to disk immediately. The default behavior is switched off due to performance considerations. pulsar=$PULSAR_HOME/bin/pulsar -nohup $pulsar $command "$1" > "$out" 2>&1 < /dev/null & +nohup $pulsar $command $1 > "$out" 2>&1 < /dev/null & Review Comment: ```suggestion nohup $pulsar $command "$@" > "$out" 2>&1 < /dev/null & ``` ## bin/pulsar-daemon: ## @@ -157,7 +157,7 @@ start () echo starting $command, logging to $logfile echo Note: Set immediateFlush to true in conf/log4j2.yaml will guarantee the logging event is flushing to disk immediately. The default behavior is switched off due to performance considerations. pulsar=$PULSAR_HOME/bin/pulsar -nohup $pulsar $command "$1" > "$out" 2>&1 < /dev/null & +nohup $pulsar $command $1 > "$out" 2>&1 < /dev/null & Review Comment: using `"$@"` is the correct thing to do. Another change is also needed. `start "$*"` should be replaced with `start "$@"` on line 219. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [fix][cli] Fix the pulsar-daemon parameter passthrough syntax [pulsar]
coderzc opened a new pull request, #22905: URL: https://github.com/apache/pulsar/pull/22905 ### Motivation Using `bin/pulsar-daemon start standalone` can't start pulsar due to parameter issue. ### Modifications replace `"$1"` with `$1` in bin/pulsar ### Verifying this change - [ ] Make sure that the change passes the CI checks. *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Does this pull request potentially affect one of the following parts: *If the box was checked, please highlight the changes* - [ ] Dependencies (add or upgrade a dependency) - [ ] The public API - [ ] The schema - [ ] The default values of configurations - [ ] The threading model - [ ] The binary protocol - [ ] The REST endpoints - [ ] The admin CLI options - [ ] The metrics - [ ] Anything that affects deployment ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` ### Matching PR in forked repository PR in forked repository: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] High GC pause cases high publish latency [pulsar]
lhotari commented on issue #22866: URL: https://github.com/apache/pulsar/issues/22866#issuecomment-2167218549 > I think so. But we might need to improve here to avoid allocating a very large `long[]` to give GC pressure. :) One alternative would be to switch to use https://github.com/lemire/RoaringBitmap/ . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] Broker became irresponsive due to too many open files error [pulsar]
lhotari commented on issue #22894: URL: https://github.com/apache/pulsar/issues/22894#issuecomment-2167216460 This comment explains one source of the problems: https://github.com/apache/pulsar/pull/22541#issuecomment-2071568113 . The problem hasn't been resolved. Namespace deletion is especially problematic, see comment https://github.com/apache/pulsar/pull/22541#issuecomment-2071621213 . /cc @mattisonchao -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix] [broker] Fix typo in PersistentTopicsBase [pulsar]
lhotari commented on PR #22904: URL: https://github.com/apache/pulsar/pull/22904#issuecomment-2167212471 Thanks for the PR @richiefanfan! This is a good way to start Pulsar contributions! There are tools such as https://github.com/crate-ci/typos which could scan the whole code base at once. We shouldn't change typos in class names and method names in public APIs without first deprecating. Those should be handled separately case-by-case. All typos in log messages or exception messages could be batched together to fewer PRs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] `status.html` can't access using 3.3.0 image [pulsar]
coderzc closed issue #22903: [Bug] `status.html` can't access using 3.3.0 image URL: https://github.com/apache/pulsar/issues/22903 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [fix] [broker] Fix typo in PersistentTopicsBase [pulsar]
richiefanfan opened a new pull request, #22904: URL: https://github.com/apache/pulsar/pull/22904 ### Motivation Fix typo problem. ### Modifications ### Documentation - [ ] `doc` - [ ] `doc-required` - [ ] `doc-not-needed` - [ ] `doc-complete` ### Matching PR in forked repository PR in forked repository: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] [Bug] `status.html` can't access using 3.3.0 image [pulsar]
coderzc opened a new issue, #22903: URL: https://github.com/apache/pulsar/issues/22903 ### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Read release policy - [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker. ### Version 3.3.0 ### Minimal reproduce step curl http://localhost:8080/status.html ### What did you expect to see? response 200 ### What did you see instead? response ``` Error 404 Not Found HTTP ERROR 404 Not Found URI:/status.html STATUS:404 MESSAGE:Not Found SERVLET:org.glassfish.jersey.servlet.ServletContainer-72eed547 https://eclipse.org/jetty;>Powered by Jetty:// 9.4.54.v20240208 ``` pulsar log ``` 2024-06-14T03:17:41,561+ [pulsar-web-48-9] WARN org.apache.pulsar.common.configuration.VipStatus - Failed to access "status.html". The service is not ready 2024-06-14T03:17:41,592+ [pulsar-web-48-9] INFO org.eclipse.jetty.server.RequestLog - 127.0.0.1 - - [14/Jun/2024:03:17:41 +] "GET /status.html HTTP/1.1" 404 508 "-" "curl/8.5.0" 65 ``` ### Anything else? _No response_ ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [Fix][broker] Limit replication rate based on bytes [pulsar]
nodece commented on PR #22674: URL: https://github.com/apache/pulsar/pull/22674#issuecomment-2167094595 /pulsarbot rerun-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix] [broker] fix unload bundle count metric. [pulsar]
thetumbled commented on PR #22895: URL: https://github.com/apache/pulsar/pull/22895#issuecomment-2167078168 /pulsarbot rerun-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix] [broker] fix unload bundle count metric. [pulsar]
thetumbled commented on PR #22895: URL: https://github.com/apache/pulsar/pull/22895#issuecomment-2167070763 /pulsarbot rerun-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch master updated: [improve][meta] Fix invalid use of drain API and race condition in closing metadata store (#22585)
This is an automated email from the ASF dual-hosted git repository.
mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new f7d35e5ddbf [improve][meta] Fix invalid use of drain API and race
condition in closing metadata store (#22585)
f7d35e5ddbf is described below
commit f7d35e5ddbfb96ef4eda636ba7808868dc56017f
Author: Lari Hotari
AuthorDate: Fri Jun 14 04:24:07 2024 +0300
[improve][meta] Fix invalid use of drain API and race condition in closing
metadata store (#22585)
Co-authored-by: Matteo Merli
---
.../batching/AbstractBatchedMetadataStore.java | 29 +++---
1 file changed, 25 insertions(+), 4 deletions(-)
diff --git
a/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/batching/AbstractBatchedMetadataStore.java
b/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/batching/AbstractBatchedMetadataStore.java
index 5b45530d2e2..4fa1c6aca0f 100644
---
a/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/batching/AbstractBatchedMetadataStore.java
+++
b/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/batching/AbstractBatchedMetadataStore.java
@@ -86,9 +86,13 @@ public abstract class AbstractBatchedMetadataStore extends
AbstractMetadataStore
// Fail all the pending items
MetadataStoreException ex =
new
MetadataStoreException.AlreadyClosedException("Metadata store is getting
closed");
-readOps.drain(op -> op.getFuture().completeExceptionally(ex));
-writeOps.drain(op -> op.getFuture().completeExceptionally(ex));
-
+MetadataOp op;
+while ((op = readOps.poll()) != null) {
+op.getFuture().completeExceptionally(ex);
+}
+while ((op = writeOps.poll()) != null) {
+op.getFuture().completeExceptionally(ex);
+}
scheduledTask.cancel(true);
}
super.close();
@@ -98,7 +102,13 @@ public abstract class AbstractBatchedMetadataStore extends
AbstractMetadataStore
private void flush() {
while (!readOps.isEmpty()) {
List ops = new ArrayList<>();
-readOps.drain(ops::add, maxOperations);
+for (int i = 0; i < maxOperations; i++) {
+MetadataOp op = readOps.poll();
+if (op == null) {
+break;
+}
+ops.add(op);
+}
internalBatchOperation(ops);
}
@@ -167,6 +177,11 @@ public abstract class AbstractBatchedMetadataStore extends
AbstractMetadataStore
}
private void enqueue(MessagePassingQueue queue, MetadataOp op)
{
+if (isClosed()) {
+MetadataStoreException ex = new
MetadataStoreException.AlreadyClosedException();
+op.getFuture().completeExceptionally(ex);
+return;
+}
if (enabled) {
if (!queue.offer(op)) {
// Execute individually if we're failing to enqueue
@@ -182,6 +197,12 @@ public abstract class AbstractBatchedMetadataStore extends
AbstractMetadataStore
}
private void internalBatchOperation(List ops) {
+if (isClosed()) {
+MetadataStoreException ex =
+new MetadataStoreException.AlreadyClosedException();
+ops.forEach(op -> op.getFuture().completeExceptionally(ex));
+return;
+}
long now = System.currentTimeMillis();
for (MetadataOp op : ops) {
this.batchMetadataStoreStats.recordOpWaiting(now - op.created());
Re: [PR] [improve][meta] Fix invalid use of drain API and race condition in closing metadata store [pulsar]
merlimat merged PR #22585: URL: https://github.com/apache/pulsar/pull/22585 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch master updated: [fix][broker] Asynchronously return brokerRegistry.lookupAsync when checking if broker is active(ExtensibleLoadManagerImpl only) (#22899)
This is an automated email from the ASF dual-hosted git repository.
heesung pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new c2702e9bc46 [fix][broker] Asynchronously return
brokerRegistry.lookupAsync when checking if broker is
active(ExtensibleLoadManagerImpl only) (#22899)
c2702e9bc46 is described below
commit c2702e9bc46c444cbc99f4b64cb453c622b56c26
Author: Heesung Sohn <103456639+heesung...@users.noreply.github.com>
AuthorDate: Thu Jun 13 12:26:40 2024 -0700
[fix][broker] Asynchronously return brokerRegistry.lookupAsync when
checking if broker is active(ExtensibleLoadManagerImpl only) (#22899)
---
.../channel/ServiceUnitStateChannelImpl.java | 84 --
.../channel/ServiceUnitStateChannelTest.java | 51 ++---
2 files changed, 89 insertions(+), 46 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java
index 069ac516551..f04734c4ad9 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java
@@ -484,7 +484,7 @@ public class ServiceUnitStateChannelImpl implements
ServiceUnitStateChannel {
String serviceUnit,
ServiceUnitState state,
Optional owner) {
-return deferGetOwnerRequest(serviceUnit)
+return dedupeGetOwnerRequest(serviceUnit)
.thenCompose(newOwner -> {
if (newOwner == null) {
return CompletableFuture.completedFuture(null);
@@ -622,7 +622,7 @@ public class ServiceUnitStateChannelImpl implements
ServiceUnitStateChannel {
}
EventType eventType = Assign;
eventCounters.get(eventType).getTotal().incrementAndGet();
-CompletableFuture getOwnerRequest =
deferGetOwnerRequest(serviceUnit);
+CompletableFuture getOwnerRequest =
dedupeGetOwnerRequest(serviceUnit);
pubAsync(serviceUnit, new ServiceUnitStateData(Assigning, broker,
getNextVersionId(serviceUnit)))
.whenComplete((__, ex) -> {
@@ -932,44 +932,54 @@ public class ServiceUnitStateChannelImpl implements
ServiceUnitStateChannel {
return broker.equals(brokerId);
}
-private CompletableFuture deferGetOwnerRequest(String serviceUnit)
{
+private CompletableFuture deferGetOwner(String serviceUnit) {
+var future = new
CompletableFuture().orTimeout(inFlightStateWaitingTimeInMillis,
+TimeUnit.MILLISECONDS)
+.exceptionally(e -> {
+var ownerAfter = getOwner(serviceUnit);
+log.warn("{} failed to wait for owner for serviceUnit:{};
Trying to "
++ "return the current owner:{}",
+brokerId, serviceUnit, ownerAfter, e);
+if (ownerAfter == null) {
+throw new IllegalStateException(e);
+}
+return ownerAfter.orElse(null);
+});
+if (debug()) {
+log.info("{} is waiting for owner for serviceUnit:{}", brokerId,
serviceUnit);
+}
+return future;
+}
+
+private CompletableFuture dedupeGetOwnerRequest(String
serviceUnit) {
var requested = new MutableObject>();
try {
-return getOwnerRequests
-.computeIfAbsent(serviceUnit, k -> {
-var ownerBefore = getOwner(serviceUnit);
-if (ownerBefore != null && ownerBefore.isPresent()) {
-// Here, we do a quick active check first with the
computeIfAbsent lock
-
brokerRegistry.lookupAsync(ownerBefore.get()).getNow(Optional.empty())
-.ifPresent(__ -> requested.setValue(
-
CompletableFuture.completedFuture(ownerBefore.get(;
-
-if (requested.getValue() != null) {
-return requested.getValue();
-}
-}
-
-
-CompletableFuture future =
-new
CompletableFuture().orTimeout(inFlightStateWaitingTimeInMillis,
-TimeUnit.MILLISECONDS)
-.exceptionally(e -> {
-var ownerAfter =
getOwner(serviceUnit);
-log.warn("{}
Re: [PR] [fix][broker] Asynchronously return brokerRegistry.lookupAsync when checking if broker is active(ExtensibleLoadManagerImpl only) [pulsar]
heesung-sn merged PR #22899: URL: https://github.com/apache/pulsar/pull/22899 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][broker] Asynchronously return brokerRegistry.lookupAsync when checking if broker is active(ExtensibleLoadManagerImpl only) [pulsar]
heesung-sn closed pull request #22899: [fix][broker] Asynchronously return brokerRegistry.lookupAsync when checking if broker is active(ExtensibleLoadManagerImpl only) URL: https://github.com/apache/pulsar/pull/22899 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [improve][misc] Replace dependencies on PositionImpl with Position interface [pulsar]
lhotari merged PR #22891: URL: https://github.com/apache/pulsar/pull/22891 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Upgrade vcpkg dependencies for the missed boost-format dependency [pulsar-client-cpp]
BewareMyPower commented on PR #427: URL: https://github.com/apache/pulsar-client-cpp/pull/427#issuecomment-2165805554 There is a bug for GCC 4.8 that it does not support [`std::align`](https://en.cppreference.com/w/cpp/memory/align), which is a standard of C++11. I sent an email to drop GCC 4.8 in the next release (see https://lists.apache.org/thread/7o8hpv1gtoffvzx053wm0ss2s9xt0795), so I will open a PR to drop the support for GCC 4.8 first and then go back to this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [improve][client]:Support custom message listener executor for specific subscription [pulsar]
AuroraTwinkle commented on PR #22861: URL: https://github.com/apache/pulsar/pull/22861#issuecomment-2165522377 > I added some further comments to simplify MessageListenerExecutor. The lifecycle is managed by the application that provides the instance and that's why there shouldn't be any details of the executor. Behind the `MessageListenerExecutor` there might be multiple different queues / thread pools. Passing the message instance will allow using the key or any other properties of the message to do decisions when it is needed, for example in an executor that uses a priority queue implementation. > > This change will also need a PIP. Please check for other PIPs on the mailing list for an example https://lists.apache.org/list.html?d...@pulsar.apache.org . You can find mailing list joining instructions at https://pulsar.apache.org/contact/ . The PIP template is at https://github.com/apache/pulsar/blob/master/pip/TEMPLATE.md . Before the PIP is accepted, we cannot merge implementation PRs. Hi!@lhotari, excuse me, I have written [PIP-359](https://github.com/apache/pulsar/pull/22902) for this, looking forward to your suggestions! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [improve][pip] PIP-359: Support custom message listener executor for specific subscription [pulsar]
AuroraTwinkle opened a new pull request, #22902: URL: https://github.com/apache/pulsar/pull/22902 ### Motivation PIP-359 Implementation PR：https://github.com/apache/pulsar/pull/22861 ### Modifications ### Verifying this change - [ ] Make sure that the change passes the CI checks. *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Does this pull request potentially affect one of the following parts: *If the box was checked, please highlight the changes* - [ ] Dependencies (add or upgrade a dependency) - [ ] The public API - [ ] The schema - [ ] The default values of configurations - [ ] The threading model - [ ] The binary protocol - [ ] The REST endpoints - [ ] The admin CLI options - [ ] The metrics - [ ] Anything that affects deployment ### Documentation - [ ] `doc` - [ ] `doc-required` - [ ] `doc-not-needed` - [ ] `doc-complete` ### Matching PR in forked repository PR in forked repository: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] [Bug] Major compaction is not recovered automatically after the disk is writable again [pulsar]
isminex opened a new issue, #22901: URL: https://github.com/apache/pulsar/issues/22901 ### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Read release policy - [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker. ### Version v2.11.2 ### Minimal reproduce step 1. Major compaction task is suspended because the disk is almost full.  2. Minor compaction task is also suspended because the disk is full-filled.  3. Disk becomes writable again.  **Problem1**: Only `fulfilledDirs` will be recovered. If the disks(dirs) are just almost full(not full-filled), there is no change their major compaction or minor compaction task can be recovered.  **Problem2**: Because the disk's usage is still not under warn threshold (0.9) when they are writable, only minor compaction task can be recovered and then the disks(dirs) are removed from `fulfilledDirs`, So there is no change their major compaction compaction task can be recovered because of `Problem1`.  3. Major compaction task can not be recovered even the disk is idle.  ### What did you expect to see? Major compaction is recoveryed automatically when the dist is idle again. ### What did you see instead? Major compaction cannot be recoveryed automatically when the dist is idle again. ### Anything else? _No response_ ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch branch-3.0 updated: [fix][broker][branch-3.0] The topic might reference a closed ledger (#22860) (#22900)
This is an automated email from the ASF dual-hosted git repository.
baodi pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 8be3e8ab7b1 [fix][broker][branch-3.0] The topic might reference a
closed ledger (#22860) (#22900)
8be3e8ab7b1 is described below
commit 8be3e8ab7b14451ea31e2a979c1b5fa2d1f993d6
Author: Baodi Shi
AuthorDate: Thu Jun 13 19:18:02 2024 +0800
[fix][broker][branch-3.0] The topic might reference a closed ledger
(#22860) (#22900)
---
.../org/apache/pulsar/broker/PulsarService.java| 5 +
.../pulsar/broker/service/BrokerService.java | 154 ++---
.../pulsar/broker/service/ReplicatorTest.java | 10 +-
.../client/api/OrphanPersistentTopicTest.java | 68 +
4 files changed, 150 insertions(+), 87 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
index 47509e9bc49..07284f8371b 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
@@ -1940,6 +1940,11 @@ public class PulsarService implements AutoCloseable,
ShutdownService {
return new BrokerService(pulsar, ioEventLoopGroup);
}
+@VisibleForTesting
+public void setTransactionExecutorProvider(TransactionBufferProvider
transactionBufferProvider) {
+this.transactionBufferProvider = transactionBufferProvider;
+}
+
public void initConfigMetadataSynchronizerIfNeeded() {
mutex.lock();
try {
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
index 3a1afb6e657..3d17aedf813 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
@@ -1044,38 +1044,38 @@ public class BrokerService implements Closeable {
return getTopic(TopicName.get(topic), createIfMissing, properties);
}
+/**
+ * Retrieves or creates a topic based on the specified parameters.
+ * 0. If disable PersistentTopics or NonPersistentTopics, it will return a
failed future with NotAllowedException.
+ * 1. If topic future exists in the cache returned directly regardless of
whether it fails or timeout.
+ * 2. If the topic metadata exists, the topic is created regardless of
{@code createIfMissing}.
+ * 3. If the topic metadata not exists, and {@code createIfMissing} is
false,
+ *returns an empty Optional in a CompletableFuture. And this empty
future not be added to the map.
+ * 4. Otherwise, use computeIfAbsent. It returns the existing topic or
creates and adds a new topicFuture.
+ *Any exceptions will remove the topicFuture from the map.
+ *
+ * @param topicName The name of the topic, potentially including partition
information.
+ * @param createIfMissing If true, creates the topic if it does not exist.
+ * @param properties Topic configuration properties used during creation.
+ * @return CompletableFuture with an Optional of the topic if found or
created, otherwise empty.
+ */
public CompletableFuture> getTopic(final TopicName
topicName, boolean createIfMissing,
Map
properties) {
try {
-CompletableFuture> topicFuture =
topics.get(topicName.toString());
-if (topicFuture != null) {
-if (topicFuture.isCompletedExceptionally()
-|| (topicFuture.isDone() &&
!topicFuture.getNow(Optional.empty()).isPresent())) {
-// Exceptional topics should be recreated.
-topics.remove(topicName.toString(), topicFuture);
-} else {
-// a non-existing topic in the cache shouldn't prevent
creating a topic
-if (createIfMissing) {
-if (topicFuture.isDone() &&
topicFuture.getNow(Optional.empty()).isPresent()) {
-return topicFuture;
-} else {
-return topicFuture.thenCompose(value -> {
-if (!value.isPresent()) {
-// retry and create topic
-return getTopic(topicName,
createIfMissing, properties);
-} else {
-// in-progress future completed
successfully
-return
CompletableFuture.completedFuture(value);
-}
-});
-
Re: [PR] [fix][broker][branch-3.0] The topic might reference a closed ledger (#22860) [pulsar]
shibd merged PR #22900: URL: https://github.com/apache/pulsar/pull/22900 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [improve][misc] Replace dependencies on PositionImpl with Position interface [pulsar]
lhotari commented on code in PR #22891:
URL: https://github.com/apache/pulsar/pull/22891#discussion_r1637917795
##
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/Position.java:
##
@@ -26,16 +27,116 @@
*/
@InterfaceAudience.LimitedPrivate
@InterfaceStability.Stable
-public interface Position {
+public interface Position extends Comparable {
+/**
+ * Get the ledger id of the entry pointed by this position.
+ *
+ * @return the ledger id
+ */
+long getLedgerId();
+
+/**
+ * Get the entry id of the entry pointed by this position.
+ *
+ * @return the entry id
+ */
+long getEntryId();
+
+/**
+ * Compare this position with another position.
+ * The comparison is first based on the ledger id, and then on the entry
id.
+ * This is implements the Comparable interface.
+ * @param that the other position to be compared.
+ * @return -1 if this position is less than the other, 0 if they are
equal, 1 if this position is greater than
+ * the other.
+ */
+default int compareTo(Position that) {
+if (getLedgerId() != that.getLedgerId()) {
+return (getLedgerId() < that.getLedgerId() ? -1 : 1);
+}
+
+if (getEntryId() != that.getEntryId()) {
+return (getEntryId() < that.getEntryId() ? -1 : 1);
+}
+
+return 0;
Review Comment:
Thanks @merlimat, nice improvement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
(pulsar) branch branch-3.3 updated: [fix] Fix cherry-pick for #22892
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a commit to branch branch-3.3 in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/branch-3.3 by this push: new e75f6ba1d42 [fix] Fix cherry-pick for #22892 e75f6ba1d42 is described below commit e75f6ba1d4273ccf1a17c5e118ba010b4bd7e8ec Author: Lari Hotari AuthorDate: Thu Jun 13 12:34:16 2024 +0300 [fix] Fix cherry-pick for #22892 - version in jetcd-core-shaded/pom.xml needs to match project version --- jetcd-core-shaded/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jetcd-core-shaded/pom.xml b/jetcd-core-shaded/pom.xml index d8819a1148a..8fdcc7c8098 100644 --- a/jetcd-core-shaded/pom.xml +++ b/jetcd-core-shaded/pom.xml @@ -26,7 +26,7 @@ org.apache.pulsar pulsar -3.4.0-SNAPSHOT +3.3.1-SNAPSHOT jetcd-core-shaded
Re: [PR] [improve][misc] Replace dependencies on PositionImpl with Position interface [pulsar]
lhotari commented on PR #22891: URL: https://github.com/apache/pulsar/pull/22891#issuecomment-2165149124 > This change is so massive that I think docs are in fact required to explain all of the changes in a TL/DR manner. @dave2wave The reason why this change is massive is mainly due to the PositionImpl -> Position/PositionFactory changes. The PR description already contains the explanation in a TL/DR manner in "Modifications". Anything missing? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) 02/02: [improve][misc] Upgrade to Netty 4.1.111.Final and switch to use grpc-netty-shaded (#22892)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit cdda561115ad2beb5122a056007580e4bcf0bce1
Author: Lari Hotari
AuthorDate: Thu Jun 13 01:24:04 2024 +0300
[improve][misc] Upgrade to Netty 4.1.111.Final and switch to use
grpc-netty-shaded (#22892)
(cherry picked from commit 75d7e557d84bf2cca2ec791dfe8479b8a6df7875)
---
distribution/server/pom.xml| 13 ++
distribution/server/src/assemble/LICENSE.bin.txt | 50 +++---
distribution/shell/src/assemble/LICENSE.bin.txt| 40 ++---
jetcd-core-shaded/pom.xml | 187 +
pom.xml| 60 ++-
pulsar-broker/pom.xml | 12 ++
pulsar-functions/instance/pom.xml | 9 +-
pulsar-metadata/pom.xml| 11 +-
.../pulsar/metadata/impl/EtcdMetadataStore.java| 6 +-
9 files changed, 329 insertions(+), 59 deletions(-)
diff --git a/distribution/server/pom.xml b/distribution/server/pom.xml
index d7f032cd54e..1b40a75432a 100644
--- a/distribution/server/pom.xml
+++ b/distribution/server/pom.xml
@@ -40,6 +40,19 @@
${project.version}
+
+ ${project.groupId}
+ pulsar-metadata
+ ${project.version}
+
+
+
+ ${project.groupId}
+ jetcd-core-shaded
+ ${project.version}
+ shaded
+
+
${project.groupId}
pulsar-docs-tools
diff --git a/distribution/server/src/assemble/LICENSE.bin.txt
b/distribution/server/src/assemble/LICENSE.bin.txt
index 9a510643f8e..e65c1b2ea8f 100644
--- a/distribution/server/src/assemble/LICENSE.bin.txt
+++ b/distribution/server/src/assemble/LICENSE.bin.txt
@@ -289,27 +289,27 @@ The Apache Software License, Version 2.0
- org.apache.commons-commons-lang3-3.11.jar
- org.apache.commons-commons-text-1.10.0.jar
* Netty
-- io.netty-netty-buffer-4.1.108.Final.jar
-- io.netty-netty-codec-4.1.108.Final.jar
-- io.netty-netty-codec-dns-4.1.108.Final.jar
-- io.netty-netty-codec-http-4.1.108.Final.jar
-- io.netty-netty-codec-http2-4.1.108.Final.jar
-- io.netty-netty-codec-socks-4.1.108.Final.jar
-- io.netty-netty-codec-haproxy-4.1.108.Final.jar
-- io.netty-netty-common-4.1.108.Final.jar
-- io.netty-netty-handler-4.1.108.Final.jar
-- io.netty-netty-handler-proxy-4.1.108.Final.jar
-- io.netty-netty-resolver-4.1.108.Final.jar
-- io.netty-netty-resolver-dns-4.1.108.Final.jar
-- io.netty-netty-resolver-dns-classes-macos-4.1.108.Final.jar
-- io.netty-netty-resolver-dns-native-macos-4.1.108.Final-osx-aarch_64.jar
-- io.netty-netty-resolver-dns-native-macos-4.1.108.Final-osx-x86_64.jar
-- io.netty-netty-transport-4.1.108.Final.jar
-- io.netty-netty-transport-classes-epoll-4.1.108.Final.jar
-- io.netty-netty-transport-native-epoll-4.1.108.Final-linux-aarch_64.jar
-- io.netty-netty-transport-native-epoll-4.1.108.Final-linux-x86_64.jar
-- io.netty-netty-transport-native-unix-common-4.1.108.Final.jar
--
io.netty-netty-transport-native-unix-common-4.1.108.Final-linux-x86_64.jar
+- io.netty-netty-buffer-4.1.111.Final.jar
+- io.netty-netty-codec-4.1.111.Final.jar
+- io.netty-netty-codec-dns-4.1.111.Final.jar
+- io.netty-netty-codec-http-4.1.111.Final.jar
+- io.netty-netty-codec-http2-4.1.111.Final.jar
+- io.netty-netty-codec-socks-4.1.111.Final.jar
+- io.netty-netty-codec-haproxy-4.1.111.Final.jar
+- io.netty-netty-common-4.1.111.Final.jar
+- io.netty-netty-handler-4.1.111.Final.jar
+- io.netty-netty-handler-proxy-4.1.111.Final.jar
+- io.netty-netty-resolver-4.1.111.Final.jar
+- io.netty-netty-resolver-dns-4.1.111.Final.jar
+- io.netty-netty-resolver-dns-classes-macos-4.1.111.Final.jar
+- io.netty-netty-resolver-dns-native-macos-4.1.111.Final-osx-aarch_64.jar
+- io.netty-netty-resolver-dns-native-macos-4.1.111.Final-osx-x86_64.jar
+- io.netty-netty-transport-4.1.111.Final.jar
+- io.netty-netty-transport-classes-epoll-4.1.111.Final.jar
+- io.netty-netty-transport-native-epoll-4.1.111.Final-linux-aarch_64.jar
+- io.netty-netty-transport-native-epoll-4.1.111.Final-linux-x86_64.jar
+- io.netty-netty-transport-native-unix-common-4.1.111.Final.jar
+-
io.netty-netty-transport-native-unix-common-4.1.111.Final-linux-x86_64.jar
- io.netty-netty-tcnative-boringssl-static-2.0.65.Final.jar
- io.netty-netty-tcnative-boringssl-static-2.0.65.Final-linux-aarch_64.jar
- io.netty-netty-tcnative-boringssl-static-2.0.65.Final-linux-x86_64.jar
@@ -424,7 +424,6 @@ The Apache Software License, Version 2.0
- io.grpc-grpc-auth-1.55.3.jar
- io.grpc-grpc-context-1.55.3.jar
- io.grpc-grpc-core-1.55.3.jar
-- io.grpc-grpc-netty-1.55.3.jar
- io.grpc-grpc-protobuf-1.55.3.jar
-
(pulsar) 01/02: [improve][broker] Include runtime dependencies in server distribution (#22001)
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a commit to branch branch-3.2 in repository https://gitbox.apache.org/repos/asf/pulsar.git commit f35b51d445d255d1e5c397e0c50bbb7b6828dd5c Author: Dragos Misca AuthorDate: Wed Jan 31 10:01:44 2024 -0800 [improve][broker] Include runtime dependencies in server distribution (#22001) (cherry picked from commit 57025bc11913680f7aac26ab42399ea8a6fccc05) --- distribution/server/src/assemble/LICENSE.bin.txt | 2 +- distribution/server/src/assemble/bin.xml | 9 ++--- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/distribution/server/src/assemble/LICENSE.bin.txt b/distribution/server/src/assemble/LICENSE.bin.txt index 7f8410b0472..9a510643f8e 100644 --- a/distribution/server/src/assemble/LICENSE.bin.txt +++ b/distribution/server/src/assemble/LICENSE.bin.txt @@ -506,6 +506,7 @@ The Apache Software License, Version 2.0 - io.reactivex.rxjava3-rxjava-3.0.1.jar * RoaringBitmap - org.roaringbitmap-RoaringBitmap-0.9.44.jar +- org.roaringbitmap-shims-0.9.44.jar BSD 3-clause "New" or "Revised" License * Google auth library @@ -538,7 +539,6 @@ Protocol Buffers License CDDL-1.1 -- ../licenses/LICENSE-CDDL-1.1.txt * Java Annotations API -- javax.annotation-javax.annotation-api-1.3.2.jar - com.sun.activation-javax.activation-1.2.0.jar - javax.xml.bind-jaxb-api-2.3.1.jar * Java Servlet API -- javax.servlet-javax.servlet-api-3.1.0.jar diff --git a/distribution/server/src/assemble/bin.xml b/distribution/server/src/assemble/bin.xml index 949c2657069..4dfec015c0e 100644 --- a/distribution/server/src/assemble/bin.xml +++ b/distribution/server/src/assemble/bin.xml @@ -110,7 +110,7 @@ lib false - compile + runtime false @@ -119,12 +119,15 @@ org.apache.pulsar:pulsar-functions-runtime-all -org.projectlombok:lombok - org.apache.pulsar:pulsar-functions-api-examples *:tar.gz + +org.codehaus.mojo:animal-sniffer-annotations +com.google.android:annotations + +net.java.dev.jna:jna
(pulsar) branch branch-3.2 updated (7914163babb -> cdda561115a)
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a change to branch branch-3.2 in repository https://gitbox.apache.org/repos/asf/pulsar.git from 7914163babb [improve] Upgrade IPAddress to 5.5.0 (#22886) new f35b51d445d [improve][broker] Include runtime dependencies in server distribution (#22001) new cdda561115a [improve][misc] Upgrade to Netty 4.1.111.Final and switch to use grpc-netty-shaded (#22892) The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: distribution/server/pom.xml| 13 ++ distribution/server/src/assemble/LICENSE.bin.txt | 52 +++--- distribution/server/src/assemble/bin.xml | 9 +- distribution/shell/src/assemble/LICENSE.bin.txt| 40 ++--- jetcd-core-shaded/pom.xml | 187 + pom.xml| 60 ++- pulsar-broker/pom.xml | 12 ++ pulsar-functions/instance/pom.xml | 9 +- pulsar-metadata/pom.xml| 11 +- .../pulsar/metadata/impl/EtcdMetadataStore.java| 6 +- 10 files changed, 336 insertions(+), 63 deletions(-) create mode 100644 jetcd-core-shaded/pom.xml
(pulsar) branch branch-3.3 updated: [fix][broker] The topic might reference a closed ledger (#22860)
This is an automated email from the ASF dual-hosted git repository.
baodi pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new 8ecd94b84b8 [fix][broker] The topic might reference a closed ledger
(#22860)
8ecd94b84b8 is described below
commit 8ecd94b84b81119e98965353e8e82cb3f6ec898a
Author: Baodi Shi
AuthorDate: Thu Jun 13 16:49:05 2024 +0800
[fix][broker] The topic might reference a closed ledger (#22860)
(cherry picked from commit a91a172b4ee6d8b974a3fa905e435975557fcc57)
---
.../org/apache/pulsar/broker/PulsarService.java| 5 +
.../pulsar/broker/service/BrokerService.java | 155 ++---
.../pulsar/broker/service/ReplicatorTest.java | 10 +-
.../client/api/OrphanPersistentTopicTest.java | 68 +
4 files changed, 151 insertions(+), 87 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
index 6482ead1f5a..b7deaa0c79c 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
@@ -1922,6 +1922,11 @@ public class PulsarService implements AutoCloseable,
ShutdownService {
return new BrokerService(pulsar, ioEventLoopGroup);
}
+@VisibleForTesting
+public void setTransactionExecutorProvider(TransactionBufferProvider
transactionBufferProvider) {
+this.transactionBufferProvider = transactionBufferProvider;
+}
+
private CompactionServiceFactory loadCompactionServiceFactory() {
String compactionServiceFactoryClassName =
config.getCompactionServiceFactoryClassName();
var compactionServiceFactory =
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
index 0cba1647031..371df1c0056 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
@@ -999,38 +999,38 @@ public class BrokerService implements Closeable {
return getTopic(TopicName.get(topic), createIfMissing, properties);
}
+/**
+ * Retrieves or creates a topic based on the specified parameters.
+ * 0. If disable PersistentTopics or NonPersistentTopics, it will return a
failed future with NotAllowedException.
+ * 1. If topic future exists in the cache returned directly regardless of
whether it fails or timeout.
+ * 2. If the topic metadata exists, the topic is created regardless of
{@code createIfMissing}.
+ * 3. If the topic metadata not exists, and {@code createIfMissing} is
false,
+ *returns an empty Optional in a CompletableFuture. And this empty
future not be added to the map.
+ * 4. Otherwise, use computeIfAbsent. It returns the existing topic or
creates and adds a new topicFuture.
+ *Any exceptions will remove the topicFuture from the map.
+ *
+ * @param topicName The name of the topic, potentially including partition
information.
+ * @param createIfMissing If true, creates the topic if it does not exist.
+ * @param properties Topic configuration properties used during creation.
+ * @return CompletableFuture with an Optional of the topic if found or
created, otherwise empty.
+ */
public CompletableFuture> getTopic(final TopicName
topicName, boolean createIfMissing,
Map
properties) {
try {
-CompletableFuture> topicFuture =
topics.get(topicName.toString());
-if (topicFuture != null) {
-if (topicFuture.isCompletedExceptionally()
-|| (topicFuture.isDone() &&
!topicFuture.getNow(Optional.empty()).isPresent())) {
-// Exceptional topics should be recreated.
-topics.remove(topicName.toString(), topicFuture);
-} else {
-// a non-existing topic in the cache shouldn't prevent
creating a topic
-if (createIfMissing) {
-if (topicFuture.isDone() &&
topicFuture.getNow(Optional.empty()).isPresent()) {
-return topicFuture;
-} else {
-return topicFuture.thenCompose(value -> {
-if (!value.isPresent()) {
-// retry and create topic
-return getTopic(topicName,
createIfMissing, properties);
-} else {
-// in-progress future completed
successfully
-
Re: [PR] [fix][broker][branch-3.0] The topic might reference a closed ledger (#22860) [pulsar]
shibd closed pull request #22900: [fix][broker][branch-3.0] The topic might reference a closed ledger (#22860) URL: https://github.com/apache/pulsar/pull/22900 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [fix][broker][branch-3.0] The topic might reference a closed ledger (#22860) [pulsar]
shibd opened a new pull request, #22900: URL: https://github.com/apache/pulsar/pull/22900 (cherry picked from commit a91a172b4ee6d8b974a3fa905e435975557fcc57) ### Motivation cherry picked from #22860 ### Modifications ### Verifying this change - [ ] Make sure that the change passes the CI checks. *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Does this pull request potentially affect one of the following parts: *If the box was checked, please highlight the changes* - [ ] Dependencies (add or upgrade a dependency) - [ ] The public API - [ ] The schema - [ ] The default values of configurations - [ ] The threading model - [ ] The binary protocol - [ ] The REST endpoints - [ ] The admin CLI options - [ ] The metrics - [ ] Anything that affects deployment ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` ### Matching PR in forked repository PR in forked repository: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix] [broker] Messages lost on the remote cluster when using topic level replication [pulsar]
nodece commented on code in PR #22890:
URL: https://github.com/apache/pulsar/pull/22890#discussion_r1637853829
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java:
##
@@ -553,6 +547,21 @@ private void createPersistentSubscriptions() {
checkReplicatedSubscriptionControllerState();
}
+private CompletableFuture removeOrphanReplicationCursors() {
+List> futures = new ArrayList<>();
+List replicationClusters =
topicPolicies.getReplicationClusters().get();
+for (ManagedCursor cursor : ledger.getCursors()) {
Review Comment:
```suggestion
if (replicationClusters == null || replicationClusters.isEmpty() ||
(replicationClusters.size() == 1
&&
replicationClusters.contains(brokerService.pulsar().getConfiguration().getClusterName(
{
// no replication clusters.
return CompletableFuture.completedFuture(null);
}
for (ManagedCursor cursor : ledger.getCursors()) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [fix] [broker] Messages lost on the remote cluster when using topic level replication [pulsar]
nodece commented on code in PR #22890:
URL: https://github.com/apache/pulsar/pull/22890#discussion_r1637855265
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java:
##
@@ -553,6 +547,21 @@ private void createPersistentSubscriptions() {
checkReplicatedSubscriptionControllerState();
}
+private CompletableFuture removeOrphanReplicationCursors() {
+List> futures = new ArrayList<>();
+List replicationClusters =
topicPolicies.getReplicationClusters().get();
+for (ManagedCursor cursor : ledger.getCursors()) {
+if (cursor.getName().startsWith(replicatorPrefix)) {
+String remoteCluster =
PersistentReplicator.getRemoteCluster(cursor.getName());
+if (!replicationClusters.contains(remoteCluster)) {
+log.warn("Remove the replicator because the cluster '{}'
does not exist", remoteCluster);
Review Comment:
Please feel free to remove this warning. The removeReplicator will print an
info log when a replicator will be removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [fix][broker] The topic might reference a closed ledger [pulsar]
shibd merged PR #22860: URL: https://github.com/apache/pulsar/pull/22860 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][cursor] Fix enable maxPosition cursor will read in deap loop [pulsar]
TakaHiR07 commented on PR #14286: URL: https://github.com/apache/pulsar/pull/14286#issuecomment-2165014007 > It looks the issue fixed by #14667, closing... feel free to reopen if this PR still ongoing @dao-jun This issue is not fix. I test again in version-3.0.5 and this issue reproduce. This pr still need to be merged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix] [broker] Messages lost on the remote cluster when using topic level replication [pulsar]
nodece commented on code in PR #22890:
URL: https://github.com/apache/pulsar/pull/22890#discussion_r1637853829
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java:
##
@@ -553,6 +547,21 @@ private void createPersistentSubscriptions() {
checkReplicatedSubscriptionControllerState();
}
+private CompletableFuture removeOrphanReplicationCursors() {
+List> futures = new ArrayList<>();
+List replicationClusters =
topicPolicies.getReplicationClusters().get();
+for (ManagedCursor cursor : ledger.getCursors()) {
Review Comment:
```suggestion
if (replicationClusters == null || replicationClusters.isEmpty() ||
(replicationClusters.size() == 1
&&
replicationClusters.contains(brokerService.pulsar().getConfiguration().getClusterName(
{
// no replication clusters.
return CompletableFuture.completedFuture(null);
}
for (ManagedCursor cursor : ledger.getCursors()) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [I] [Bug][txn] exclusive subscription would lead to high cpu usage when do tailing-read [pulsar]
TakaHiR07 commented on issue #19169: URL: https://github.com/apache/pulsar/issues/19169#issuecomment-2165031292 > oh, sure. I will make #14286 merge quickly @congbobo184 Is you willing to continue working on this pr ? I test transaction in pulsar-3.0 and issue still exist. And I want to cherry-pick this pr to pulsar-3.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch master updated: [fix][broker] The topic might reference a closed ledger (#22860)
This is an automated email from the ASF dual-hosted git repository.
baodi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new a91a172b4ee [fix][broker] The topic might reference a closed ledger
(#22860)
a91a172b4ee is described below
commit a91a172b4ee6d8b974a3fa905e435975557fcc57
Author: Baodi Shi
AuthorDate: Thu Jun 13 16:49:05 2024 +0800
[fix][broker] The topic might reference a closed ledger (#22860)
---
.../org/apache/pulsar/broker/PulsarService.java| 5 +
.../pulsar/broker/service/BrokerService.java | 155 ++---
.../pulsar/broker/service/ReplicatorTest.java | 10 +-
.../client/api/OrphanPersistentTopicTest.java | 68 +
4 files changed, 151 insertions(+), 87 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
index 2e9f9dc6b01..6cbc99e2cf4 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
@@ -1964,6 +1964,11 @@ public class PulsarService implements AutoCloseable,
ShutdownService {
return new BrokerService(pulsar, ioEventLoopGroup);
}
+@VisibleForTesting
+public void setTransactionExecutorProvider(TransactionBufferProvider
transactionBufferProvider) {
+this.transactionBufferProvider = transactionBufferProvider;
+}
+
private CompactionServiceFactory loadCompactionServiceFactory() {
String compactionServiceFactoryClassName =
config.getCompactionServiceFactoryClassName();
var compactionServiceFactory =
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
index 9a08578ee40..82d7fad3874 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
@@ -1001,38 +1001,38 @@ public class BrokerService implements Closeable {
return getTopic(TopicName.get(topic), createIfMissing, properties);
}
+/**
+ * Retrieves or creates a topic based on the specified parameters.
+ * 0. If disable PersistentTopics or NonPersistentTopics, it will return a
failed future with NotAllowedException.
+ * 1. If topic future exists in the cache returned directly regardless of
whether it fails or timeout.
+ * 2. If the topic metadata exists, the topic is created regardless of
{@code createIfMissing}.
+ * 3. If the topic metadata not exists, and {@code createIfMissing} is
false,
+ *returns an empty Optional in a CompletableFuture. And this empty
future not be added to the map.
+ * 4. Otherwise, use computeIfAbsent. It returns the existing topic or
creates and adds a new topicFuture.
+ *Any exceptions will remove the topicFuture from the map.
+ *
+ * @param topicName The name of the topic, potentially including partition
information.
+ * @param createIfMissing If true, creates the topic if it does not exist.
+ * @param properties Topic configuration properties used during creation.
+ * @return CompletableFuture with an Optional of the topic if found or
created, otherwise empty.
+ */
public CompletableFuture> getTopic(final TopicName
topicName, boolean createIfMissing,
Map
properties) {
try {
-CompletableFuture> topicFuture =
topics.get(topicName.toString());
-if (topicFuture != null) {
-if (topicFuture.isCompletedExceptionally()
-|| (topicFuture.isDone() &&
!topicFuture.getNow(Optional.empty()).isPresent())) {
-// Exceptional topics should be recreated.
-topics.remove(topicName.toString(), topicFuture);
-} else {
-// a non-existing topic in the cache shouldn't prevent
creating a topic
-if (createIfMissing) {
-if (topicFuture.isDone() &&
topicFuture.getNow(Optional.empty()).isPresent()) {
-return topicFuture;
-} else {
-return topicFuture.thenCompose(value -> {
-if (!value.isPresent()) {
-// retry and create topic
-return getTopic(topicName,
createIfMissing, properties);
-} else {
-// in-progress future completed
successfully
-return
CompletableFuture.completedFuture(value);
-
(pulsar) 05/05: [improve][misc] Upgrade to Netty 4.1.111.Final and switch to use grpc-netty-shaded (#22892)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 9ced6341204bd486de7e837f6e0f3bf84febb17f
Author: Lari Hotari
AuthorDate: Thu Jun 13 01:24:04 2024 +0300
[improve][misc] Upgrade to Netty 4.1.111.Final and switch to use
grpc-netty-shaded (#22892)
(cherry picked from commit 75d7e557d84bf2cca2ec791dfe8479b8a6df7875)
(cherry picked from commit a982d7b2efb3aad456c9c0ec921c8f7bbcb48ab3)
# Conflicts:
# distribution/server/pom.xml
---
buildtools/pom.xml | 2 +-
distribution/server/pom.xml| 13 ++
distribution/server/src/assemble/LICENSE.bin.txt | 50 +++---
distribution/shell/src/assemble/LICENSE.bin.txt| 40 ++---
jetcd-core-shaded/pom.xml | 187 +
pom.xml| 60 ++-
pulsar-broker/pom.xml | 12 ++
pulsar-functions/instance/pom.xml | 9 +-
pulsar-metadata/pom.xml| 11 +-
.../pulsar/metadata/impl/EtcdMetadataStore.java| 6 +-
pulsar-sql/presto-distribution/LICENSE | 53 +++---
.../presto-distribution/src/assembly/assembly.xml | 3 +
.../presto-pulsar-plugin/src/assembly/assembly.xml | 3 +
src/check-binary-license.sh| 2 +-
14 files changed, 360 insertions(+), 91 deletions(-)
diff --git a/buildtools/pom.xml b/buildtools/pom.xml
index 8d46e280a6e..2ffb7ece6a6 100644
--- a/buildtools/pom.xml
+++ b/buildtools/pom.xml
@@ -47,7 +47,7 @@
4.1
10.14.2
3.1.2
-4.1.108.Final
+4.1.111.Final
4.2.3
32.1.1-jre
1.10.12
diff --git a/distribution/server/pom.xml b/distribution/server/pom.xml
index d3c8cacc4ff..07f5b8bafba 100644
--- a/distribution/server/pom.xml
+++ b/distribution/server/pom.xml
@@ -46,6 +46,19 @@
${project.version}
+
+ ${project.groupId}
+ pulsar-metadata
+ ${project.version}
+
+
+
+ ${project.groupId}
+ jetcd-core-shaded
+ ${project.version}
+ shaded
+
+
${project.groupId}
pulsar-proxy
diff --git a/distribution/server/src/assemble/LICENSE.bin.txt
b/distribution/server/src/assemble/LICENSE.bin.txt
index 8d218656fce..ab726ba37e9 100644
--- a/distribution/server/src/assemble/LICENSE.bin.txt
+++ b/distribution/server/src/assemble/LICENSE.bin.txt
@@ -289,27 +289,27 @@ The Apache Software License, Version 2.0
- org.apache.commons-commons-lang3-3.11.jar
- org.apache.commons-commons-text-1.10.0.jar
* Netty
-- io.netty-netty-buffer-4.1.108.Final.jar
-- io.netty-netty-codec-4.1.108.Final.jar
-- io.netty-netty-codec-dns-4.1.108.Final.jar
-- io.netty-netty-codec-http-4.1.108.Final.jar
-- io.netty-netty-codec-http2-4.1.108.Final.jar
-- io.netty-netty-codec-socks-4.1.108.Final.jar
-- io.netty-netty-codec-haproxy-4.1.108.Final.jar
-- io.netty-netty-common-4.1.108.Final.jar
-- io.netty-netty-handler-4.1.108.Final.jar
-- io.netty-netty-handler-proxy-4.1.108.Final.jar
-- io.netty-netty-resolver-4.1.108.Final.jar
-- io.netty-netty-resolver-dns-4.1.108.Final.jar
-- io.netty-netty-resolver-dns-classes-macos-4.1.108.Final.jar
-- io.netty-netty-resolver-dns-native-macos-4.1.108.Final-osx-aarch_64.jar
-- io.netty-netty-resolver-dns-native-macos-4.1.108.Final-osx-x86_64.jar
-- io.netty-netty-transport-4.1.108.Final.jar
-- io.netty-netty-transport-classes-epoll-4.1.108.Final.jar
-- io.netty-netty-transport-native-epoll-4.1.108.Final-linux-aarch_64.jar
-- io.netty-netty-transport-native-epoll-4.1.108.Final-linux-x86_64.jar
-- io.netty-netty-transport-native-unix-common-4.1.108.Final.jar
--
io.netty-netty-transport-native-unix-common-4.1.108.Final-linux-x86_64.jar
+- io.netty-netty-buffer-4.1.111.Final.jar
+- io.netty-netty-codec-4.1.111.Final.jar
+- io.netty-netty-codec-dns-4.1.111.Final.jar
+- io.netty-netty-codec-http-4.1.111.Final.jar
+- io.netty-netty-codec-http2-4.1.111.Final.jar
+- io.netty-netty-codec-socks-4.1.111.Final.jar
+- io.netty-netty-codec-haproxy-4.1.111.Final.jar
+- io.netty-netty-common-4.1.111.Final.jar
+- io.netty-netty-handler-4.1.111.Final.jar
+- io.netty-netty-handler-proxy-4.1.111.Final.jar
+- io.netty-netty-resolver-4.1.111.Final.jar
+- io.netty-netty-resolver-dns-4.1.111.Final.jar
+- io.netty-netty-resolver-dns-classes-macos-4.1.111.Final.jar
+- io.netty-netty-resolver-dns-native-macos-4.1.111.Final-osx-aarch_64.jar
+- io.netty-netty-resolver-dns-native-macos-4.1.111.Final-osx-x86_64.jar
+- io.netty-netty-transport-4.1.111.Final.jar
+- io.netty-netty-transport-classes-epoll-4.1.111.Final.jar
+- io.netty-netty-transport-native-epoll-4.1.111.Final-linux-aarch_64.jar
+-
(pulsar) 04/05: [fix] Bump io.airlift:aircompressor from 0.20 to 0.27 (#22819)
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a commit to branch branch-3.1 in repository https://gitbox.apache.org/repos/asf/pulsar.git commit c9f888f618aab852cd7bb555f86e9e339f0338b6 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Mon Jun 3 11:07:24 2024 -0700 [fix] Bump io.airlift:aircompressor from 0.20 to 0.27 (#22819) Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Matteo Merli (cherry picked from commit 1a7b72fc21961ffefa3c56af41709e6afbf13bc5) --- distribution/server/src/assemble/LICENSE.bin.txt | 2 +- distribution/shell/src/assemble/LICENSE.bin.txt | 2 +- pom.xml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/distribution/server/src/assemble/LICENSE.bin.txt b/distribution/server/src/assemble/LICENSE.bin.txt index ba7272b5235..8d218656fce 100644 --- a/distribution/server/src/assemble/LICENSE.bin.txt +++ b/distribution/server/src/assemble/LICENSE.bin.txt @@ -378,7 +378,7 @@ The Apache Software License, Version 2.0 - org.apache.httpcomponents-httpclient-4.5.13.jar - org.apache.httpcomponents-httpcore-4.4.15.jar * AirCompressor -- io.airlift-aircompressor-0.20.jar +- io.airlift-aircompressor-0.27.jar * AsyncHttpClient - org.asynchttpclient-async-http-client-2.12.1.jar - org.asynchttpclient-async-http-client-netty-utils-2.12.1.jar diff --git a/distribution/shell/src/assemble/LICENSE.bin.txt b/distribution/shell/src/assemble/LICENSE.bin.txt index f5aaf164f6d..992f1eab090 100644 --- a/distribution/shell/src/assemble/LICENSE.bin.txt +++ b/distribution/shell/src/assemble/LICENSE.bin.txt @@ -391,7 +391,7 @@ The Apache Software License, Version 2.0 - cpu-affinity-4.16.5.jar - circe-checksum-4.16.5.jar * AirCompressor - - aircompressor-0.20.jar + - aircompressor-0.27.jar * AsyncHttpClient - async-http-client-2.12.1.jar - async-http-client-netty-utils-2.12.1.jar diff --git a/pom.xml b/pom.xml index 93437f7796c..5cae16566fd 100644 --- a/pom.xml +++ b/pom.xml @@ -211,7 +211,7 @@ flexible messaging model and an intuitive client API. 1.0 0.16.1 6.2.8 -0.20 +0.27 2.12.1 1.82 3.11
(pulsar) branch branch-3.1 updated (5e6287fe5da -> 9ced6341204)
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a change to branch branch-3.1 in repository https://gitbox.apache.org/repos/asf/pulsar.git from 5e6287fe5da [improve][ci] Migrate from Gradle Enterprise to Develocity (#22880) new cdd40e18852 [improve][broker] Include runtime dependencies in server distribution (#22001) new 406d0b02d11 [improve] Upgrade Jetcd to 0.7.7 and VertX to 4.5.8 (#22835) new 69ecbcd9d3c [improve][misc] Include native epoll library for Netty for arm64 (#22319) new c9f888f618a [fix] Bump io.airlift:aircompressor from 0.20 to 0.27 (#22819) new 9ced6341204 [improve][misc] Upgrade to Netty 4.1.111.Final and switch to use grpc-netty-shaded (#22892) The 5 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: buildtools/pom.xml | 2 +- distribution/server/pom.xml| 13 ++ distribution/server/src/assemble/LICENSE.bin.txt | 66 distribution/server/src/assemble/bin.xml | 9 +- distribution/shell/src/assemble/LICENSE.bin.txt| 41 ++--- jetcd-core-shaded/pom.xml | 187 + pom.xml| 85 +- pulsar-broker/pom.xml | 12 ++ pulsar-common/pom.xml | 6 + pulsar-functions/instance/pom.xml | 9 +- pulsar-metadata/pom.xml| 11 +- .../pulsar/metadata/impl/EtcdMetadataStore.java| 6 +- pulsar-sql/presto-distribution/LICENSE | 57 +++ .../presto-distribution/src/assembly/assembly.xml | 3 + .../presto-pulsar-plugin/src/assembly/assembly.xml | 3 + src/check-binary-license.sh| 2 +- tests/integration/pom.xml | 6 + .../tests/integration/io/PulsarIOTestRunner.java | 7 +- .../integration/io/sinks/PulsarIOSinkRunner.java | 2 +- .../io/sources/PulsarIOSourceRunner.java | 2 +- .../debezium/PulsarIODebeziumSourceRunner.java | 2 +- 21 files changed, 420 insertions(+), 111 deletions(-) create mode 100644 jetcd-core-shaded/pom.xml
(pulsar) 03/05: [improve][misc] Include native epoll library for Netty for arm64 (#22319)
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a commit to branch branch-3.1 in repository https://gitbox.apache.org/repos/asf/pulsar.git commit 69ecbcd9d3c89d4dd4aaf8a7e14390829b91ee93 Author: Lari Hotari AuthorDate: Thu Mar 21 13:23:21 2024 -0700 [improve][misc] Include native epoll library for Netty for arm64 (#22319) (cherry picked from commit 24e9437ce065613fd924a74f61b620d9fdc0058b) (cherry picked from commit 22b724fd1c3eac463834a58102d667617451d453) # Conflicts: # distribution/server/src/assemble/LICENSE.bin.txt # distribution/shell/src/assemble/LICENSE.bin.txt # pulsar-sql/presto-distribution/LICENSE --- distribution/server/src/assemble/LICENSE.bin.txt | 1 + distribution/shell/src/assemble/LICENSE.bin.txt | 1 + pulsar-common/pom.xml| 6 ++ pulsar-sql/presto-distribution/LICENSE | 1 + 4 files changed, 9 insertions(+) diff --git a/distribution/server/src/assemble/LICENSE.bin.txt b/distribution/server/src/assemble/LICENSE.bin.txt index ba356456baf..ba7272b5235 100644 --- a/distribution/server/src/assemble/LICENSE.bin.txt +++ b/distribution/server/src/assemble/LICENSE.bin.txt @@ -306,6 +306,7 @@ The Apache Software License, Version 2.0 - io.netty-netty-resolver-dns-native-macos-4.1.108.Final-osx-x86_64.jar - io.netty-netty-transport-4.1.108.Final.jar - io.netty-netty-transport-classes-epoll-4.1.108.Final.jar +- io.netty-netty-transport-native-epoll-4.1.108.Final-linux-aarch_64.jar - io.netty-netty-transport-native-epoll-4.1.108.Final-linux-x86_64.jar - io.netty-netty-transport-native-unix-common-4.1.108.Final.jar - io.netty-netty-transport-native-unix-common-4.1.108.Final-linux-x86_64.jar diff --git a/distribution/shell/src/assemble/LICENSE.bin.txt b/distribution/shell/src/assemble/LICENSE.bin.txt index 96d515c2da1..f5aaf164f6d 100644 --- a/distribution/shell/src/assemble/LICENSE.bin.txt +++ b/distribution/shell/src/assemble/LICENSE.bin.txt @@ -357,6 +357,7 @@ The Apache Software License, Version 2.0 - netty-resolver-dns-4.1.108.Final.jar - netty-transport-4.1.108.Final.jar - netty-transport-classes-epoll-4.1.108.Final.jar +- netty-transport-native-epoll-4.1.108.Final-linux-aarch_64.jar - netty-transport-native-epoll-4.1.108.Final-linux-x86_64.jar - netty-transport-native-unix-common-4.1.108.Final.jar - netty-transport-native-unix-common-4.1.108.Final-linux-x86_64.jar diff --git a/pulsar-common/pom.xml b/pulsar-common/pom.xml index 0071689c133..9e85ce8dff3 100644 --- a/pulsar-common/pom.xml +++ b/pulsar-common/pom.xml @@ -99,6 +99,12 @@ linux-x86_64 + + io.netty + netty-transport-native-epoll + linux-aarch_64 + + io.netty netty-transport-native-unix-common diff --git a/pulsar-sql/presto-distribution/LICENSE b/pulsar-sql/presto-distribution/LICENSE index 982d5db850b..96aa8a2eb4f 100644 --- a/pulsar-sql/presto-distribution/LICENSE +++ b/pulsar-sql/presto-distribution/LICENSE @@ -255,6 +255,7 @@ The Apache Software License, Version 2.0 - netty-tcnative-classes-2.0.65.Final.jar - netty-transport-4.1.108.Final.jar - netty-transport-classes-epoll-4.1.108.Final.jar +- netty-transport-native-epoll-4.1.108.Final-linux-aarch_64.jar - netty-transport-native-epoll-4.1.108.Final-linux-x86_64.jar - netty-transport-native-unix-common-4.1.108.Final.jar - netty-transport-native-unix-common-4.1.108.Final-linux-x86_64.jar
(pulsar) 01/05: [improve][broker] Include runtime dependencies in server distribution (#22001)
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a commit to branch branch-3.1 in repository https://gitbox.apache.org/repos/asf/pulsar.git commit cdd40e188522bd0f152d69006111891171a3b252 Author: Dragos Misca AuthorDate: Wed Jan 31 10:01:44 2024 -0800 [improve][broker] Include runtime dependencies in server distribution (#22001) (cherry picked from commit 57025bc11913680f7aac26ab42399ea8a6fccc05) --- distribution/server/src/assemble/LICENSE.bin.txt | 2 +- distribution/server/src/assemble/bin.xml | 9 ++--- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/distribution/server/src/assemble/LICENSE.bin.txt b/distribution/server/src/assemble/LICENSE.bin.txt index 1130ef43cdf..50aa63977aa 100644 --- a/distribution/server/src/assemble/LICENSE.bin.txt +++ b/distribution/server/src/assemble/LICENSE.bin.txt @@ -504,6 +504,7 @@ The Apache Software License, Version 2.0 - io.reactivex.rxjava3-rxjava-3.0.1.jar * RoaringBitmap - org.roaringbitmap-RoaringBitmap-0.9.44.jar +- org.roaringbitmap-shims-0.9.44.jar BSD 3-clause "New" or "Revised" License * Google auth library @@ -536,7 +537,6 @@ Protocol Buffers License CDDL-1.1 -- ../licenses/LICENSE-CDDL-1.1.txt * Java Annotations API -- javax.annotation-javax.annotation-api-1.3.2.jar - com.sun.activation-javax.activation-1.2.0.jar - javax.xml.bind-jaxb-api-2.3.1.jar * Java Servlet API -- javax.servlet-javax.servlet-api-3.1.0.jar diff --git a/distribution/server/src/assemble/bin.xml b/distribution/server/src/assemble/bin.xml index 41ac24d0582..e4123f56f4f 100644 --- a/distribution/server/src/assemble/bin.xml +++ b/distribution/server/src/assemble/bin.xml @@ -126,7 +126,7 @@ lib false - compile + runtime false @@ -135,12 +135,15 @@ org.apache.pulsar:pulsar-functions-runtime-all -org.projectlombok:lombok - org.apache.pulsar:pulsar-functions-api-examples *:tar.gz + +org.codehaus.mojo:animal-sniffer-annotations +com.google.android:annotations + +net.java.dev.jna:jna
(pulsar) 02/05: [improve] Upgrade Jetcd to 0.7.7 and VertX to 4.5.8 (#22835)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 406d0b02d11eef97b842d84c44c6babe40ffaf14
Author: Matteo Merli
AuthorDate: Tue Jun 4 03:18:39 2024 -0700
[improve] Upgrade Jetcd to 0.7.7 and VertX to 4.5.8 (#22835)
(cherry picked from commit be5eb919f8c9fb4612fea74054eee8c1412b954b)
(cherry picked from commit 46b5419224c1a821ec0b9a839d0d7da57ac5f181)
---
distribution/server/src/assemble/LICENSE.bin.txt | 23 +++---
pom.xml| 23 --
pulsar-sql/presto-distribution/LICENSE | 17
tests/integration/pom.xml | 6 ++
.../tests/integration/io/PulsarIOTestRunner.java | 7 ---
.../integration/io/sinks/PulsarIOSinkRunner.java | 2 +-
.../io/sources/PulsarIOSourceRunner.java | 2 +-
.../debezium/PulsarIODebeziumSourceRunner.java | 2 +-
8 files changed, 55 insertions(+), 27 deletions(-)
diff --git a/distribution/server/src/assemble/LICENSE.bin.txt
b/distribution/server/src/assemble/LICENSE.bin.txt
index 50aa63977aa..ba356456baf 100644
--- a/distribution/server/src/assemble/LICENSE.bin.txt
+++ b/distribution/server/src/assemble/LICENSE.bin.txt
@@ -436,6 +436,7 @@ The Apache Software License, Version 2.0
- io.grpc-grpc-rls-1.55.3.jar
- io.grpc-grpc-servlet-1.55.3.jar
- io.grpc-grpc-servlet-jakarta-1.55.3.jar
+- io.grpc-grpc-util-1.60.0.jar
* Perfmark
- io.perfmark-perfmark-api-0.26.0.jar
* OpenCensus
@@ -444,7 +445,7 @@ The Apache Software License, Version 2.0
- io.opencensus-opencensus-proto-0.2.0.jar
* Jodah
- net.jodah-typetools-0.5.0.jar
-- net.jodah-failsafe-2.4.4.jar
+- dev.failsafe-failsafe-3.3.2.jar
* Byte Buddy
- net.bytebuddy-byte-buddy-1.14.12.jar
* zt-zip
@@ -476,12 +477,12 @@ The Apache Software License, Version 2.0
* JCTools - Java Concurrency Tools for the JVM
- org.jctools-jctools-core-2.1.2.jar
* Vertx
-- io.vertx-vertx-auth-common-4.3.8.jar
-- io.vertx-vertx-bridge-common-4.3.8.jar
-- io.vertx-vertx-core-4.3.8.jar
-- io.vertx-vertx-web-4.3.8.jar
-- io.vertx-vertx-web-common-4.3.8.jar
-- io.vertx-vertx-grpc-4.3.5.jar
+- io.vertx-vertx-auth-common-4.5.8.jar
+- io.vertx-vertx-bridge-common-4.5.8.jar
+- io.vertx-vertx-core-4.5.8.jar
+- io.vertx-vertx-web-4.5.8.jar
+- io.vertx-vertx-web-common-4.5.8.jar
+- io.vertx-vertx-grpc-4.5.8.jar
* Apache ZooKeeper
- org.apache.zookeeper-zookeeper-3.9.2.jar
- org.apache.zookeeper-zookeeper-jute-3.9.2.jar
@@ -494,10 +495,10 @@ The Apache Software License, Version 2.0
- com.google.auto.value-auto-value-annotations-1.9.jar
- com.google.re2j-re2j-1.6.jar
* Jetcd
-- io.etcd-jetcd-api-0.7.5.jar
-- io.etcd-jetcd-common-0.7.5.jar
-- io.etcd-jetcd-core-0.7.5.jar
-- io.etcd-jetcd-grpc-0.7.5.jar
+- io.etcd-jetcd-api-0.7.7.jar
+- io.etcd-jetcd-common-0.7.7.jar
+- io.etcd-jetcd-core-0.7.7.jar
+- io.etcd-jetcd-grpc-0.7.7.jar
* IPAddress
- com.github.seancfoley-ipaddress-5.3.3.jar
* RxJava
diff --git a/pom.xml b/pom.xml
index b4b68b99031..93437f7796c 100644
--- a/pom.xml
+++ b/pom.xml
@@ -150,7 +150,7 @@ flexible messaging model and an intuitive client
API.
2.34
1.10.50
0.16.0
-4.3.8
+4.5.8
7.9.2
1.7.32
4.4
@@ -248,13 +248,14 @@ flexible messaging model and an intuitive client
API.
5.3.27
4.5.13
4.4.15
-0.7.5
+0.7.7
2.0
1.10.12
5.3.3
3.4.3
1.5.2-3
2.0.6
+3.3.2
1.18.3
@@ -374,6 +375,12 @@ flexible messaging model and an intuitive client
API.
${mockito.version}
+
+dev.failsafe
+failsafe
+${failsafe.version}
+
+
org.apache.zookeeper
zookeeper
@@ -495,6 +502,11 @@ flexible messaging model and an intuitive client
API.
vertx-web
${vertx.version}
+
+ io.vertx
+ vertx-grpc
+ ${vertx.version}
+
org.apache.curator
@@ -593,6 +605,13 @@ flexible messaging model and an intuitive client
API.
+
+io.grpc
+grpc-util
+
+1.60.0
+
+
org.apache.bookkeeper
bookkeeper-common
diff --git a/pulsar-sql/presto-distribution/LICENSE
b/pulsar-sql/presto-distribution/LICENSE
index 03187ff3900..982d5db850b 100644
--- a/pulsar-sql/presto-distribution/LICENSE
+++ b/pulsar-sql/presto-distribution/LICENSE
@@ -271,17 +271,18 @@ The Apache Software License, Version 2.0
- grpc-protobuf-1.55.3.jar
- grpc-protobuf-lite-1.55.3.jar
- grpc-stub-1.55.3.jar
+- grpc-util-1.60.0.jar
* JEtcd
-- jetcd-api-0.7.5.jar
--
Re: [PR] [fix] [broker] Messages lost on the remote cluster when using topic level replication [pulsar]
poorbarcode commented on PR #22890: URL: https://github.com/apache/pulsar/pull/22890#issuecomment-2164867179 @nodece > BTW, the ns and topic policies are asynchronous, and we also need to use a single thread for serial updates. https://github.com/apache/pulsar/issues/21303 should be fixed first before we can handle GEO replication correctly. Correct, that is what the current PR did -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch branch-3.0 updated: [improve][misc] Upgrade to Netty 4.1.111.Final and switch to use grpc-netty-shaded (#22892)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new a982d7b2efb [improve][misc] Upgrade to Netty 4.1.111.Final and switch
to use grpc-netty-shaded (#22892)
a982d7b2efb is described below
commit a982d7b2efb3aad456c9c0ec921c8f7bbcb48ab3
Author: Lari Hotari
AuthorDate: Thu Jun 13 01:24:04 2024 +0300
[improve][misc] Upgrade to Netty 4.1.111.Final and switch to use
grpc-netty-shaded (#22892)
(cherry picked from commit 75d7e557d84bf2cca2ec791dfe8479b8a6df7875)
---
buildtools/pom.xml | 2 +-
distribution/server/pom.xml| 13 ++
distribution/server/src/assemble/LICENSE.bin.txt | 50 +++---
distribution/shell/src/assemble/LICENSE.bin.txt| 40 ++---
jetcd-core-shaded/pom.xml | 187 +
pom.xml| 60 ++-
pulsar-broker/pom.xml | 12 ++
pulsar-functions/instance/pom.xml | 9 +-
pulsar-metadata/pom.xml| 11 +-
.../pulsar/metadata/impl/EtcdMetadataStore.java| 6 +-
pulsar-sql/presto-distribution/LICENSE | 53 +++---
.../presto-distribution/src/assembly/assembly.xml | 3 +
.../presto-pulsar-plugin/src/assembly/assembly.xml | 3 +
src/check-binary-license.sh| 2 +-
14 files changed, 360 insertions(+), 91 deletions(-)
diff --git a/buildtools/pom.xml b/buildtools/pom.xml
index 3cec25625cf..e872e861cbf 100644
--- a/buildtools/pom.xml
+++ b/buildtools/pom.xml
@@ -47,7 +47,7 @@
4.1
10.14.2
3.1.2
-4.1.108.Final
+4.1.111.Final
4.2.3
32.1.1-jre
1.10.12
diff --git a/distribution/server/pom.xml b/distribution/server/pom.xml
index 27f95e0b46b..136eda902a2 100644
--- a/distribution/server/pom.xml
+++ b/distribution/server/pom.xml
@@ -40,6 +40,19 @@
${project.version}
+
+ ${project.groupId}
+ pulsar-metadata
+ ${project.version}
+
+
+
+ ${project.groupId}
+ jetcd-core-shaded
+ ${project.version}
+ shaded
+
+
${project.groupId}
pulsar-proxy
diff --git a/distribution/server/src/assemble/LICENSE.bin.txt
b/distribution/server/src/assemble/LICENSE.bin.txt
index 6de77c7beab..9d6975e4a43 100644
--- a/distribution/server/src/assemble/LICENSE.bin.txt
+++ b/distribution/server/src/assemble/LICENSE.bin.txt
@@ -289,27 +289,27 @@ The Apache Software License, Version 2.0
- org.apache.commons-commons-lang3-3.11.jar
- org.apache.commons-commons-text-1.10.0.jar
* Netty
-- io.netty-netty-buffer-4.1.108.Final.jar
-- io.netty-netty-codec-4.1.108.Final.jar
-- io.netty-netty-codec-dns-4.1.108.Final.jar
-- io.netty-netty-codec-http-4.1.108.Final.jar
-- io.netty-netty-codec-http2-4.1.108.Final.jar
-- io.netty-netty-codec-socks-4.1.108.Final.jar
-- io.netty-netty-codec-haproxy-4.1.108.Final.jar
-- io.netty-netty-common-4.1.108.Final.jar
-- io.netty-netty-handler-4.1.108.Final.jar
-- io.netty-netty-handler-proxy-4.1.108.Final.jar
-- io.netty-netty-resolver-4.1.108.Final.jar
-- io.netty-netty-resolver-dns-4.1.108.Final.jar
-- io.netty-netty-resolver-dns-classes-macos-4.1.108.Final.jar
-- io.netty-netty-resolver-dns-native-macos-4.1.108.Final-osx-aarch_64.jar
-- io.netty-netty-resolver-dns-native-macos-4.1.108.Final-osx-x86_64.jar
-- io.netty-netty-transport-4.1.108.Final.jar
-- io.netty-netty-transport-classes-epoll-4.1.108.Final.jar
-- io.netty-netty-transport-native-epoll-4.1.108.Final-linux-aarch_64.jar
-- io.netty-netty-transport-native-epoll-4.1.108.Final-linux-x86_64.jar
-- io.netty-netty-transport-native-unix-common-4.1.108.Final.jar
--
io.netty-netty-transport-native-unix-common-4.1.108.Final-linux-x86_64.jar
+- io.netty-netty-buffer-4.1.111.Final.jar
+- io.netty-netty-codec-4.1.111.Final.jar
+- io.netty-netty-codec-dns-4.1.111.Final.jar
+- io.netty-netty-codec-http-4.1.111.Final.jar
+- io.netty-netty-codec-http2-4.1.111.Final.jar
+- io.netty-netty-codec-socks-4.1.111.Final.jar
+- io.netty-netty-codec-haproxy-4.1.111.Final.jar
+- io.netty-netty-common-4.1.111.Final.jar
+- io.netty-netty-handler-4.1.111.Final.jar
+- io.netty-netty-handler-proxy-4.1.111.Final.jar
+- io.netty-netty-resolver-4.1.111.Final.jar
+- io.netty-netty-resolver-dns-4.1.111.Final.jar
+- io.netty-netty-resolver-dns-classes-macos-4.1.111.Final.jar
+- io.netty-netty-resolver-dns-native-macos-4.1.111.Final-osx-aarch_64.jar
+- io.netty-netty-resolver-dns-native-macos-4.1.111.Final-osx-x86_64.jar
+- io.netty-netty-transport-4.1.111.Final.jar
+- io.netty-netty-transport-classes-epoll-4.1.111.Final.jar
+
