(pulsar) branch dependabot/go_modules/pulsar-function-go/examples/golang.org/x/net-0.23.0 deleted (was 2c5fd167005)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/pulsar-function-go/examples/golang.org/x/net-0.23.0 in repository https://gitbox.apache.org/repos/asf/pulsar.git was 2c5fd167005 Bump golang.org/x/net in /pulsar-function-go/examples The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(pulsar) branch dependabot/go_modules/pulsar-function-go/golang.org/x/net-0.23.0 deleted (was d44618c702a)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/pulsar-function-go/golang.org/x/net-0.23.0 in repository https://gitbox.apache.org/repos/asf/pulsar.git was d44618c702a Go mod tidy The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
Re: [PR] [fix] Bump golang.org/x/net from 0.17.0 to 0.23.0 in /pulsar-function-go [pulsar]
merlimat merged PR #22540: URL: https://github.com/apache/pulsar/pull/22540 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix] Bump golang.org/x/net from 0.17.0 to 0.23.0 in /pulsar-function-go/examples [pulsar]
dependabot[bot] closed pull request #22539: [fix] Bump golang.org/x/net from 0.17.0 to 0.23.0 in /pulsar-function-go/examples URL: https://github.com/apache/pulsar/pull/22539 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix] Bump golang.org/x/net from 0.17.0 to 0.23.0 in /pulsar-function-go/examples [pulsar]
dependabot[bot] commented on PR #22539: URL: https://github.com/apache/pulsar/pull/22539#issuecomment-2067515040 Looks like golang.org/x/net is up-to-date now, so this is no longer needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch master updated: [fix] Bump golang.org/x/net from 0.17.0 to 0.23.0 in /pulsar-function-go (#22540)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new 21647a1fc69 [fix] Bump golang.org/x/net from 0.17.0 to 0.23.0 in /pulsar-function-go (#22540) 21647a1fc69 is described below commit 21647a1fc69ff46e65b6eaa37dd6d435e9f8eaef Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Fri Apr 19 19:12:34 2024 -0700 [fix] Bump golang.org/x/net from 0.17.0 to 0.23.0 in /pulsar-function-go (#22540) Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Matteo Merli --- pulsar-function-go/examples/go.mod | 8 pulsar-function-go/examples/go.sum | 16 pulsar-function-go/go.mod | 8 pulsar-function-go/go.sum | 16 4 files changed, 24 insertions(+), 24 deletions(-) diff --git a/pulsar-function-go/examples/go.mod b/pulsar-function-go/examples/go.mod index 31e1cc7769b..59e695f5a33 100644 --- a/pulsar-function-go/examples/go.mod +++ b/pulsar-function-go/examples/go.mod @@ -42,11 +42,11 @@ require ( github.com/spaolacci/murmur3 v1.1.0 // indirect github.com/stretchr/testify v1.8.4 // indirect go.uber.org/atomic v1.7.0 // indirect - golang.org/x/crypto v0.17.0 // indirect - golang.org/x/net v0.17.0 // indirect + golang.org/x/crypto v0.21.0 // indirect + golang.org/x/net v0.23.0 // indirect golang.org/x/oauth2 v0.13.0 // indirect - golang.org/x/sys v0.15.0 // indirect - golang.org/x/term v0.15.0 // indirect + golang.org/x/sys v0.18.0 // indirect + golang.org/x/term v0.18.0 // indirect golang.org/x/text v0.14.0 // indirect google.golang.org/appengine v1.6.8 // indirect google.golang.org/genproto/googleapis/rpc v0.0.0-20231002182017-d307bd883b97 // indirect diff --git a/pulsar-function-go/examples/go.sum b/pulsar-function-go/examples/go.sum index 5d2429673f0..85390cf32e5 100644 --- a/pulsar-function-go/examples/go.sum +++ b/pulsar-function-go/examples/go.sum @@ -393,8 +393,8 @@ golang.org/x/crypto v0.0.0-20190820162420-60c769a6c586/go.mod h1:yigFU9vqHzYiE8U golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= -golang.org/x/crypto v0.17.0 h1:r8bRNjWL3GshPW3gkd+RpvzWrZAwPS49OmTGZ/uhM4k= -golang.org/x/crypto v0.17.0/go.mod h1:gCAAfMLgwOJRpTjQ2zCCt2OcSfYMTeZVSRtQlPC7Nq4= +golang.org/x/crypto v0.21.0 h1:X31++rzVUdKhX5sWmSOFZxx8UW/ldWx55cbf08iNAMA= +golang.org/x/crypto v0.21.0/go.mod h1:0BP7YvVV9gBbVKyeTG0Gyn+gZm94bibOW5BjDEYAOMs= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8= @@ -473,8 +473,8 @@ golang.org/x/net v0.0.0-20210316092652-d523dce5a7f4/go.mod h1:RBQZq4jEuRlivfhVLd golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM= golang.org/x/net v0.0.0-20210726213435-c6fcb2dbf985/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= -golang.org/x/net v0.17.0 h1:pVaXccu2ozPjCXewfr1S7xza/zcXTity9cCdXQYSjIM= -golang.org/x/net v0.17.0/go.mod h1:NxSsAGuq816PNPmqtQdLE42eU2Fs7NoRIZrHJAlaCOE= +golang.org/x/net v0.23.0 h1:7EYJ93RZ9vYSZAIb2x3lnuvqO5zneoD6IvWjuhfxjTs= +golang.org/x/net v0.23.0/go.mod h1:JKghWKKOSdJwpW2GEx0Ja7fmaKnMsbu+MWVZTokSYmg= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= @@ -559,12 +559,12 @@ golang.org/x/sys v0.0.0-20210603081109-ebe580a85c40/go.mod h1:oPkhp1MJrh7nUepCBc golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.15.0 h1:h48lPFYpsTvQJZF4EKyI4aLHaev3CxivZmv7yZig9pc= -golang.org/x/sys v0.15.0/go.mod
Re: [PR] [fix][meta] Use ForkJoinPool.commonPool to handle Metadata operations [pulsar]
heesung-sn commented on PR #22542: URL: https://github.com/apache/pulsar/pull/22542#issuecomment-2067507781 Although ForkJoinPool can grow very large(max 32767) -- it is less likely exhausted (Should be more than the number of caller threads), I agree that we should not chain the "heavy operation" (in this case, metadata sync calls) after the metadata operation. I will fix the caller code not to trigger the metadata sync call, instead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][meta] Use ForkJoinPool.commonPool to handle Metadata operations [pulsar]
heesung-sn closed pull request #22542: [fix][meta] Use ForkJoinPool.commonPool to handle Metadata operations URL: https://github.com/apache/pulsar/pull/22542 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch dependabot/go_modules/pulsar-function-go/golang.org/x/net-0.23.0 updated (e29291a9a36 -> d44618c702a)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a change to branch dependabot/go_modules/pulsar-function-go/golang.org/x/net-0.23.0 in repository https://gitbox.apache.org/repos/asf/pulsar.git from e29291a9a36 Bump golang.org/x/net from 0.17.0 to 0.23.0 in /pulsar-function-go add d44618c702a Go mod tidy No new revisions were added by this update. Summary of changes: pulsar-function-go/examples/go.mod | 8 pulsar-function-go/examples/go.sum | 16 2 files changed, 12 insertions(+), 12 deletions(-)
[I] Cleanup synchronous call in resources component [pulsar]
mattisonchao opened a new issue, #22544: URL: https://github.com/apache/pulsar/issues/22544 ### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Motivation We should clean up all the synchronous call-in resources to avoid someone using them to block the metadata thread. FYI: https://github.com/apache/pulsar/pull/22542 ### Solution _No response_ ### Alternatives _No response_ ### Anything else? _No response_ ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][meta] Use ForkJoinPool.commonPool to handle Metadata operations [pulsar]
mattisonchao commented on PR #22542: URL: https://github.com/apache/pulsar/pull/22542#issuecomment-2067421577 Questions: 1. This looks like just wanna mitigate the issue to the common pool. we should avoid blocking calls for metadata. 2. Did you think about multi-thread will change the request order? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [improve][client] Add connectionsPerHost and connectionMaxIdleSeconds to PulsarAdminBuilder [pulsar]
merlimat commented on code in PR #22541:
URL: https://github.com/apache/pulsar/pull/22541#discussion_r1573069445
##
pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/internal/PulsarAdminBuilderImpl.java:
##
@@ -47,6 +47,7 @@ public PulsarAdmin build() throws PulsarClientException {
public PulsarAdminBuilderImpl() {
this.conf = new ClientConfigurationData();
+this.conf.setConnectionsPerBroker(16);
Review Comment:
Couldn't the default be part of `ClientConfigurationData` constructor?
##
pulsar-client-admin-api/src/main/java/org/apache/pulsar/client/admin/PulsarAdminBuilder.java:
##
@@ -336,4 +336,30 @@ PulsarAdminBuilder authentication(String
authPluginClassName, Map
+ * By default, the connection pool maintains up to 16 connections to a
single host. This method allows you to
+ * modify this default behavior and limit the number of connections.
+ *
+ * This setting can be useful in scenarios where you want to limit the
resources used by the client library,
+ * or control the level of parallelism for operations so that a single
client does not overwhelm
+ * the Pulsar cluster with too many concurrent connections.
+ *
+ * @param connectionsPerHost the maximum number of connections to
establish per host. Set to <= 0 to disable
+ * the limit.
+ * @return the PulsarAdminBuilder instance, allowing for method chaining
+ */
+PulsarAdminBuilder connectionsPerHost(int connectionsPerHost);
Review Comment:
maybe `maxConnectionsPerHost()` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [fix][meta] Use ForkJoinPool.commonPool to handle Metadata operations [pulsar]
merlimat commented on code in PR #22542:
URL: https://github.com/apache/pulsar/pull/22542#discussion_r1573064947
##
pulsar-metadata/src/test/java/org/apache/pulsar/metadata/MetadataStoreTest.java:
##
@@ -480,6 +481,19 @@ public void testThreadSwitchOfZkMetadataStore(boolean
hasSynchronizer, boolean e
verify.run();
return null;
}).join();
+// blocking chaining call
+store.get(prefix + "/d1").thenApply((ignore) -> {
+try {
+verify.run();
+return store.get(prefix + "/e1").thenApply((ignore2)
-> {
+verify.run();
+return null;
+}).get();
Review Comment:
The problem here is a mix between sync & async calls.
eg: blocking in a callback thread.
Using ForkJoinPool, it's only moving the problem to a different place. It
will be able to avoid deadlocking on a simple 1:1 situation, though when the
ForkJoinPool threads count is maxed, we will still be subject to deadlock.
No matter how we handle this, it's going to be problematic, hence we always
have had to be careful in avoiding blocking calls to metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[I] PostgreSQL sink connector doesn't persist message to table [pulsar]
alexandrebrilhante opened a new issue, #22543:
URL: https://github.com/apache/pulsar/issues/22543
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues)
and found nothing similar.
### Read release policy
- [X] I understand that unsupported versions don't get bug fixes. I will
attempt to reproduce the issue on a supported version of Pulsar client and
Pulsar broker.
### Version
OS: macOS Sonoma 14.4.1
Java: OpenJDK 17.0.11
Pulsar: 3.2.1
### Minimal reproduce step
Example detailed
[here](https://pulsar.apache.org/docs/next/io-quickstart/#connect-pulsar-to-postgresql)
seems outdated. I've followed every step but still can't see new records in
PostgreSQL. For comparison, there's seems to be no issue when switching to
[Cassandra](https://pulsar.apache.org/docs/next/io-quickstart/#connect-pulsar-to-cassandra)
with the same schema and producer setup. I've tried with both local and
dockerized Postgres databases.
`pulsar-postgres-jdbc-sink.yaml`
```yaml
configs:
userName: "postgres"
password: "postgres"
jdbcUrl: "jdbc:postgresql://localhost:5432/postgres"
tableName: "pulsar_postgres_jdbc_sink"
```
`schema`
```json
{
"type": "AVRO",
"schema":
"{\"type\":\"record\",\"name\":\"test\",\"fields\":[{\"name\":\"id\",\"type\":[\"null\",\"int\"]},{\"name\":\"name\",\"type\":[\"null\",\"string\"]}]}",
"properties": {}
}
```
`main.rs` - 127.0.0.1: sends dummy data e.g. `{"id" 1, "name"
"abcdefg"}` which `main` then sends to Pulsar.
```rust
use pulsar::{producer::ProducerOptions, Pulsar, TokioExecutor};
use tokio::{io::AsyncReadExt, net::TcpListener, sync::mpsc};
#[tokio::main]
async fn main() {
let addr: = "pulsar://localhost:6650";
let pulsar: Pulsar<_> = Pulsar::builder(addr, TokioExecutor)
.build()
.await
.expect("Failed to connect to Pulsar...");
let topic_name: = "persistent://public/default/test_cassandra";
let mut producer: pulsar::Producer = pulsar
.producer()
.with_topic(topic_name)
.with_name("producer")
.with_options(ProducerOptions {
batch_size: Some(4),
..Default::default()
})
.build()
.await
.expect("Failed to create producer...");
let (tx, mut rx) = mpsc::channel(100);
let _producer_task: tokio::task::JoinHandle<()> = tokio::spawn(async
move {
while let Some(message) = rx.recv().await {
match producer.send(message).await {
Ok(_) => println!("Message sent to Pulsar..."),
Err(e) => eprintln!("Failed to send message to Pulsar; err =
{:?}...", e),
}
}
});
let listener: TcpListener = TcpListener::bind("127.0.0.1:")
.await
.expect("Failed to bind to address...");
loop {
let (mut socket, _addr) = listener
.accept()
.await
.expect("Failed to accept connection...");
let tx: mpsc::Sender = tx.clone();
tokio::spawn(async move {
let mut buf: [u8; 1024] = [0; 1024];
loop {
let n: usize = match socket.read( buf).await {
Ok(n) if n == 0 => return,
Ok(n) => n,
Err(e) => {
eprintln!("Failed to read from socket; err =
{:?}...", e);
return;
}
};
let message =
String::from_utf8_lossy([0..n]).to_string();
if tx.send(message).await.is_err() {
eprintln!("Failed to send message to channel...");
return;
}
}
});
}
}
```
Complete setup:
```bash
bin/pulsar standalone
bin/pulsar-admin schemas upload pulsar-postgres-jdbc-sink-topic -f
$PWD/pulsar/connectors/schema
bin/pulsar-admin sinks create \
--archive $PWD/pulsar/connectors/pulsar-io-jdbc-postgres-3.2.2.nar \
--inputs pulsar-postgres-jdbc-sink-topic \
--name pulsar-postgres-jdbc-sink \
--sink-config-file $PWD/pulsar/connectors/pulsar-postgres-jdbc-sink.yaml
\
--parallelism 1
cargo build --release && cargo run --release
```
### What did you expect to see?
PostgreSQL table `pulsar_postgres_jdbc_sink` being populated in real-time.
### What did you see instead?
PostgreSQL table `pulsar_postgres_jdbc_sink` is empty although Pulsar is the
producing the message properly.
### Anything else?
No issues when inspecting the sink or the topic. Pulsar is able to
Re: [I] Consumer able to receive message which is not matching the regex pattern [pulsar]
visortelle commented on issue #22529:
URL: https://github.com/apache/pulsar/issues/22529#issuecomment-2067251332
@ragaur-tibco I fixed your code.
Code:
```java
package b;
import org.apache.pulsar.client.admin.PulsarAdminException;
import org.apache.pulsar.client.api.*;
import java.util.List;
import java.util.concurrent.TimeUnit;
import java.util.regex.Pattern;
public class App {
private static final String SERVICE_URL = "pulsar://localhost:6650";
private static final String SUBSCRIPTION_NAME = "your-subscription";
public static void main(String[] args) throws Exception {
PulsarClient pulsarClient = PulsarClient.builder()
.serviceUrl(SERVICE_URL)
.build();
Producer producerA = pulsarClient.newProducer(Schema.STRING)
.topic("non-persistent://my-tenant/new-name/topic-non-1")
.enableBatching(false).create();
Producer producerB = pulsarClient.newProducer(Schema.STRING)
.topic("persistent://my-tenant/new-name/topic-pers-1")
.enableBatching(false).create();
Pattern allTopicsPattern = Pattern.compile("my-tenant/new-name/.*");
Consumer allTopicsConsumer = pulsarClient.newConsumer()
.topicsPattern(allTopicsPattern)
.subscriptionName(SUBSCRIPTION_NAME)
.subscriptionTopicsMode(RegexSubscriptionMode.AllTopics)
.subscribe();
producerA.send("=from topic
non-persistent://my-tenant/new-name/topic-non-1 ");
producerB.send("=from topic
persistent://my-tenant/new-name/topic-pers-1 ");
while (true) {
Message message = allTopicsConsumer.receive();
System.out.println("Received message from topic " +
message.getTopicName()
+ ": " + new String(message.getValue()));
allTopicsConsumer.acknowledge(message);
}
}
}
```
Logs:
```
a mvn exec:java
[INFO] Scanning for projects...
[INFO]
[INFO] < c:a
>-
[INFO] Building a 1.0-SNAPSHOT
[INFO] [ jar
]-
[INFO]
[INFO] --- exec-maven-plugin:3.2.0:java (default-cli) @ a ---
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
Received message from topic non-persistent://my-tenant/new-name/topic-non-1:
=from topic non-persistent://my-tenant/new-name/topic-non-1
Received message from topic persistent://my-tenant/new-name/topic-pers-1:
=from topic persistent://my-tenant/new-name/topic-pers-1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [fix][meta] Use ForkJoinPool.commonPool to handle Metadata operations [pulsar]
lhotari commented on PR #22542: URL: https://github.com/apache/pulsar/pull/22542#issuecomment-2067142518 I didn't review in detail yet. I think something like this has come up before. This might have an impact on correctness since ordering guarantees would be lost. It's worth postponing merging the PR until there's a review from @merlimat . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Consumer able to receive message which is not matching the regex pattern [pulsar]
visortelle commented on issue #22529: URL: https://github.com/apache/pulsar/issues/22529#issuecomment-2067135042 @ragaur-tibco the current behavior is correct. See my comment here: https://github.com/apache/pulsar/issues/22527#issuecomment-2067133567 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Topic creation not working properly [pulsar]
visortelle commented on issue #22527:
URL: https://github.com/apache/pulsar/issues/22527#issuecomment-2067133567
@ragaur-tibco it seems we both were wrong here and it works as expected.
I'm stupidly bad at Java concurrency, so I rewrote it in ZIO which is
simpler to understand for me.
It seems like the consumer simply doesn't have time to update the list of
topics.
Corrected code:
```scala
topics = Vector(
"persistent://new-tenant/new-namespace/topic-a",
"persistent://new-tenant/new-namespace/topic-b",
"non-persistent://new-tenant/new-namespace/topic-c",
"non-persistent://new-tenant/new-namespace/topic-d"
)
numMessagesPerTopic = 10
// Thread-safe counter
numMessagesReceivedRef <- Ref.make(0)
_ <- ZIO.attempt {
// Cleanup
pulsarAdmin.topics.getList("new-tenant/new-namespace").asScala
.foreach(pulsarAdmin.topics.delete(_, true))
pulsarAdmin.topics.getPartitionedTopicList("new-tenant/new-namespace").asScala
.foreach(pulsarAdmin.topics.deletePartitionedTopic(_, true))
}
consumer <- ZIO.attempt {
pulsarClient.newConsumer()
.topicsPattern("new-tenant/new-namespace/.*".r.pattern)
.subscriptionName("new-subscription")
.patternAutoDiscoveryPeriod(100, TimeUnit.MILLISECONDS)
.subscriptionTopicsMode(RegexSubscriptionMode.AllTopics)
.subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
.subscribe()
}
// Consume messages in background
consumeInBackgroundFib <- (for {
isMessageReceived <- ZIO.attempt {
Option(consumer.receive(1, TimeUnit.SECONDS)) match
case None => false
case Some(msg) =>
println(s"Received: ${msg.getValue.mkString(",")}. From
topic: ${msg.getTopicName}")
consumer.acknowledge(msg.getMessageId)
true
}
_ <- numMessagesReceivedRef.update(_ + 1).when(isMessageReceived)
} yield ())
.forever // like `while true`
.fork // Run in background
producers <- ZIO.attempt {
topics.map(topic => pulsarClient.newProducer.topic(topic).create())
}
// Wait for the expected number of consumers
_ <- ZIO.attempt {
// Cast consumer to PatternMultiTopicsConsumerImpl
// that has extra pattern-related methods
val numConsumers = consumer
.asInstanceOf[PatternMultiTopicsConsumerImpl[Array[Byte]]]
.getConsumers
.size
if numConsumers != topics.size
then throw new Exception(s"Expected $topics.size consumers, but got
$numConsumers")
}
.retry(Schedule.exponential(10.millis))
.timeoutFail(new Exception("Consumers weren't created in
time"))(10.seconds)
_ <- ZIO.attempt {
for (i <- 0 until numMessagesPerTopic)
producers.foreach(producer => producer.sendAsync(Array(i.toByte)))
}
// Wait for all messages are be received
_ <- (for {
numMessagesReceived <- numMessagesReceivedRef.get
_ <- ZIO.attempt {
if numMessagesReceived != topics.size * numMessagesPerTopic
then throw new Exception(s"Expected ${topics.size *
numMessagesPerTopic} messages, but got $numMessagesReceived")
}
} yield ())
.retry(Schedule.spaced(250.millis))
.timeoutFail(new Exception("Messages weren't received in
time"))(10.seconds)
numMessagesReceived <- numMessagesReceivedRef.get
_ <- ZIO.logInfo(s"Messages received: $numMessagesReceived")
_ <- consumeInBackgroundFib.join
```
Logs:
```
22:56:51.304 [pulsar-client-io-3-3] INFO
o.a.p.c.i.ProducerStatsRecorderImpl - Starting Pulsar producer perf with
config:
{"topicName":"persistent://new-tenant/new-namespace/topic-a","producerName":null,"sendTimeoutMs":3,"blockIfQueueFull":false,"maxPendingMessages":0,"maxPendingMessagesAcrossPartitions":0,"messageRoutingMode":"RoundRobinPartition","hashingScheme":"JavaStringHash","cryptoFailureAction":"FAIL","batchingMaxPublishDelayMicros":1000,"batchingPartitionSwitchFrequencyByPublishDelay":10,"batchingMaxMessages":1000,"batchingMaxBytes":131072,"batchingEnabled":true,"chunkingEnabled":false,"chunkMaxMessageSize":-1,"encryptionKeys":[],"compressionType":"NONE","initialSequenceId":null,"autoUpdatePartitions":true,"autoUpdatePartitionsIntervalSeconds":60,"multiSchema":true,"accessMode":"Shared","lazyStartPartitionedProducers":false,"properties":{},"initialSubscriptionName":null}
22:56:51.382 [pulsar-client-io-3-3] INFO
o.a.p.c.i.ProducerStatsRecorderImpl - Pulsar client config:
[PR] [fix][broker] Use ForkJoinPool.commonPool to handle Metadata operations [pulsar]
heesung-sn opened a new pull request, #22542: URL: https://github.com/apache/pulsar/pull/22542 ### Motivation Observed that the metadata store executor thread can be blocked when handling chained future operations (e.g. while running handleGetResult). Especially, when the chained future operation needs to wait for another metadata access, this can block the metadata get/put operations, because the flush() will never be called(because the executor is blocked now). ### Modifications - Use ForkJoinPool.commonPool threads to execute the metadata operations, instead of the metadata executor thread. ### Verifying this change - [x] Make sure that the change passes the CI checks. ### Does this pull request potentially affect one of the following parts: *If the box was checked, please highlight the changes* - [ ] Dependencies (add or upgrade a dependency) - [ ] The public API - [ ] The schema - [ ] The default values of configurations - [ ] The threading model - [ ] The binary protocol - [ ] The REST endpoints - [ ] The admin CLI options - [ ] The metrics - [ ] Anything that affects deployment ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` ### Matching PR in forked repository PR in forked repository: https://github.com/heesung-sn/pulsar/pull/65 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][broker] Fix broken topic policy implementation compatibility with old pulsar version [pulsar]
rdhabalia merged PR #22535: URL: https://github.com/apache/pulsar/pull/22535 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch master updated: [fix][broker] Fix broken topic policy implementation compatibility with old pulsar version (#22535)
This is an automated email from the ASF dual-hosted git repository. rdhabalia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new 59daac64c21 [fix][broker] Fix broken topic policy implementation compatibility with old pulsar version (#22535) 59daac64c21 is described below commit 59daac64c210f539e733f883edad09d08333aa62 Author: Rajan Dhabalia AuthorDate: Fri Apr 19 10:30:55 2024 -0700 [fix][broker] Fix broken topic policy implementation compatibility with old pulsar version (#22535) --- .../pulsar/broker/service/AbstractTopic.java | 52 +- ...kerInternalClientConfigurationOverrideTest.java | 42 - 2 files changed, 72 insertions(+), 22 deletions(-) diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java index e772486fcc6..44a4ca42cea 100644 --- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java +++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java @@ -220,13 +220,16 @@ public abstract class AbstractTopic implements Topic, TopicPolicyListener this.topicPolicies.getBackLogQuotaMap().get(type).updateTopicValue( data.getBackLogQuotaMap() == null ? null : data.getBackLogQuotaMap().get(type.toString(; - topicPolicies.getTopicMaxMessageSize().updateTopicValue(data.getMaxMessageSize()); - topicPolicies.getMessageTTLInSeconds().updateTopicValue(data.getMessageTTLInSeconds()); + topicPolicies.getTopicMaxMessageSize().updateTopicValue(normalizeValue(data.getMaxMessageSize())); + topicPolicies.getMessageTTLInSeconds().updateTopicValue(normalizeValue(data.getMessageTTLInSeconds())); topicPolicies.getPublishRate().updateTopicValue(PublishRate.normalize(data.getPublishRate())); topicPolicies.getDelayedDeliveryEnabled().updateTopicValue(data.getDelayedDeliveryEnabled()); topicPolicies.getReplicatorDispatchRate().updateTopicValue( @@ -268,15 +271,19 @@ public abstract class AbstractTopic implements Topic, TopicPolicyListener(CollectionUtils.emptyIfNull(namespacePolicies.replication_clusters))); topicPolicies.getMaxUnackedMessagesOnConsumer() - .updateNamespaceValue(namespacePolicies.max_unacked_messages_per_consumer); + .updateNamespaceValue(normalizeValue(namespacePolicies.max_unacked_messages_per_consumer)); topicPolicies.getMaxUnackedMessagesOnSubscription() - .updateNamespaceValue(namespacePolicies.max_unacked_messages_per_subscription); - topicPolicies.getMessageTTLInSeconds().updateNamespaceValue(namespacePolicies.message_ttl_in_seconds); - topicPolicies.getMaxSubscriptionsPerTopic().updateNamespaceValue(namespacePolicies.max_subscriptions_per_topic); - topicPolicies.getMaxProducersPerTopic().updateNamespaceValue(namespacePolicies.max_producers_per_topic); - topicPolicies.getMaxConsumerPerTopic().updateNamespaceValue(namespacePolicies.max_consumers_per_topic); + .updateNamespaceValue(normalizeValue(namespacePolicies.max_unacked_messages_per_subscription)); +topicPolicies.getMessageTTLInSeconds() + .updateNamespaceValue(normalizeValue(namespacePolicies.message_ttl_in_seconds)); +topicPolicies.getMaxSubscriptionsPerTopic() + .updateNamespaceValue(normalizeValue(namespacePolicies.max_subscriptions_per_topic)); +topicPolicies.getMaxProducersPerTopic() + .updateNamespaceValue(normalizeValue(namespacePolicies.max_producers_per_topic)); +topicPolicies.getMaxConsumerPerTopic() + .updateNamespaceValue(normalizeValue(namespacePolicies.max_consumers_per_topic)); topicPolicies.getMaxConsumersPerSubscription() - .updateNamespaceValue(namespacePolicies.max_consumers_per_subscription); + .updateNamespaceValue(normalizeValue(namespacePolicies.max_consumers_per_subscription)); topicPolicies.getInactiveTopicPolicies().updateNamespaceValue(namespacePolicies.inactive_topic_policies); topicPolicies.getDeduplicationEnabled().updateNamespaceValue(namespacePolicies.deduplicationEnabled); topicPolicies.getDeduplicationSnapshotIntervalSeconds().updateNamespaceValue( @@ -312,6 +319,10 @@ public abstract class AbstractTopic implements Topic, TopicPolicyListener producer = pulsarClient.newProducer() +.topic(topic).create(); +PersistentTopic topicRef = (PersistentTopic) pulsar.getBrokerService().getTopicReference(topic).get(); + assertEquals(topicRef.topicPolicies.getMaxUnackedMessagesOnSubscription().get(), +conf.getMaxUnackedMessagesPerSubscription()); +
Re: [I] [Bug] Broker is not blocking the dispatch even after reaching max unack-message limit per subscription [pulsar]
rdhabalia closed issue #22534: [Bug] Broker is not blocking the dispatch even after reaching max unack-message limit per subscription URL: https://github.com/apache/pulsar/issues/22534 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][broker] upgrade jclouds 2.5.0 -> 2.6.0 [pulsar]
lhotari commented on PR #0: URL: https://github.com/apache/pulsar/pull/0#issuecomment-2066834669 > @lhotari Thanks for fixing this. We should be able to test it, but maybe not for a couple more weeks. @pgier do you have any test results to share about JClouds 2.6.0 ? I'm thinking of cherry-picking to maintenance branches after there's more confirmation that JClouds 2.6.0 works well with Pulsar. /cc @dave2wave @eolivelli @nicoloboschi -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] We are leaving old releases in the ASF distribution area. Please cleanup. [pulsar]
lhotari commented on issue #22486: URL: https://github.com/apache/pulsar/issues/22486#issuecomment-2066758989 I cleaned up even more. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
svn commit: r68658 - in /release/pulsar: pulsar-2.10.6/ pulsar-2.11.4/ pulsar-2.8.4/ pulsar-2.9.5/ pulsar-3.0.4/ pulsar-3.1.3/ pulsar-3.2.2/ pulsar-adapters-2.11.0/ pulsar-adapters-2.8.0/ pulsar-clien
Author: lhotari Date: Fri Apr 19 14:53:50 2024 New Revision: 68658 Log: Delete archived Pulsar releases Removed: release/pulsar/pulsar-2.10.6/ release/pulsar/pulsar-2.11.4/ release/pulsar/pulsar-2.8.4/ release/pulsar/pulsar-2.9.5/ release/pulsar/pulsar-3.0.4/ release/pulsar/pulsar-3.1.3/ release/pulsar/pulsar-3.2.2/ release/pulsar/pulsar-adapters-2.11.0/ release/pulsar/pulsar-adapters-2.8.0/ release/pulsar/pulsar-client-cpp-3.0.0/ release/pulsar/pulsar-client-cpp-3.1.2/ release/pulsar/pulsar-client-cpp-3.2.0/ release/pulsar/pulsar-client-cpp-3.3.0/ release/pulsar/pulsar-client-cpp-3.4.2/ release/pulsar/pulsar-client-cpp-3.5.0/ release/pulsar/pulsar-client-go-0.10.0/ release/pulsar/pulsar-client-go-0.11.1/ release/pulsar/pulsar-client-go-0.8.1/ release/pulsar/pulsar-client-go-0.9.0/ release/pulsar/pulsar-client-node/pulsar-client-node-1.10.0/ release/pulsar/pulsar-client-node/pulsar-client-node-1.7.0/ release/pulsar/pulsar-client-node/pulsar-client-node-1.8.0/ release/pulsar/pulsar-client-node/pulsar-client-node-1.8.1/ release/pulsar/pulsar-client-node/pulsar-client-node-1.8.2/ release/pulsar/pulsar-client-node/pulsar-client-node-1.9.0/ release/pulsar/pulsar-client-python-3.0.0/ release/pulsar/pulsar-client-python-3.1.0/ release/pulsar/pulsar-client-python-3.2.0/ release/pulsar/pulsar-client-python-3.3.0/ release/pulsar/pulsar-client-python-3.4.0/ release/pulsar/pulsar-client-python-3.5.0/ release/pulsar/pulsar-client-reactive-0.1.0/ release/pulsar/pulsar-client-reactive-0.2.0/ release/pulsar/pulsar-client-reactive-0.3.0/ release/pulsar/pulsar-client-reactive-0.4.0/ release/pulsar/pulsar-client-reactive-0.5.3/ release/pulsar/pulsar-dotpulsar-3.0.0/ release/pulsar/pulsar-dotpulsar-3.1.2/ release/pulsar/pulsar-manager/pulsar-manager-0.3.0/
(pulsar-site) branch main updated: Use https://archive.apache.org/dist instead of https://downloads.apache.org/
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/pulsar-site.git The following commit(s) were added to refs/heads/main by this push: new 9cd115664fbd Use https://archive.apache.org/dist instead of https://downloads.apache.org/ 9cd115664fbd is described below commit 9cd115664fbd5538eaf0cd6a63f80df9a9a9d23f Author: Lari Hotari AuthorDate: Fri Apr 19 17:38:27 2024 +0300 Use https://archive.apache.org/dist instead of https://downloads.apache.org/ - so that we can delete the helm-chart releases from https://dist.apache.org/repos/dist/release/pulsar/helm-chart/ --- static/charts/index.yaml | 10 +- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/static/charts/index.yaml b/static/charts/index.yaml index 6c5b32b9b7d2..f864d01015ec 100644 --- a/static/charts/index.yaml +++ b/static/charts/index.yaml @@ -39,7 +39,7 @@ entries: - https://github.com/apache/pulsar - https://github.com/apache/pulsar-helm-chart urls: -- https://downloads.apache.org/pulsar/helm-chart/3.3.1/pulsar-3.3.1.tgz +- https://archive.apache.org/dist/pulsar/helm-chart/3.3.1/pulsar-3.3.1.tgz version: 3.3.1 - apiVersion: v2 appVersion: 3.0.2 @@ -62,7 +62,7 @@ entries: - https://github.com/apache/pulsar - https://github.com/apache/pulsar-helm-chart urls: -- https://downloads.apache.org/pulsar/helm-chart/3.3.0/pulsar-3.3.0.tgz +- https://archive.apache.org/dist/pulsar/helm-chart/3.3.0/pulsar-3.3.0.tgz version: 3.3.0 - apiVersion: v2 appVersion: 3.0.2 @@ -84,7 +84,7 @@ entries: - https://github.com/apache/pulsar - https://github.com/apache/pulsar-helm-chart urls: -- https://downloads.apache.org/pulsar/helm-chart/3.2.0/pulsar-3.2.0.tgz +- https://archive.apache.org/dist/pulsar/helm-chart/3.2.0/pulsar-3.2.0.tgz version: 3.2.0 - apiVersion: v2 appVersion: 3.0.2 @@ -106,7 +106,7 @@ entries: - https://github.com/apache/pulsar - https://github.com/apache/pulsar-helm-chart urls: -- https://downloads.apache.org/pulsar/helm-chart/3.1.0/pulsar-3.1.0.tgz +- https://archive.apache.org/dist/pulsar/helm-chart/3.1.0/pulsar-3.1.0.tgz version: 3.1.0 - apiVersion: v2 appVersion: 2.10.2 @@ -128,7 +128,7 @@ entries: - https://github.com/apache/pulsar - https://github.com/apache/pulsar-helm-chart urls: -- https://downloads.apache.org/pulsar/helm-chart/3.0.0/pulsar-3.0.0.tgz +- https://archive.apache.org/dist/pulsar/helm-chart/3.0.0/pulsar-3.0.0.tgz version: 3.0.0 - apiVersion: v2 appVersion: 2.9.3
Re: [I] Latest chart version: 3.4.0 missing in the Apache helm repo index [pulsar-helm-chart]
lhotari commented on issue #490: URL: https://github.com/apache/pulsar-helm-chart/issues/490#issuecomment-205832 > The details for this chart version are missing in the helm repo index https://pulsar.apache.org/charts/index.yaml Would appreciate any guidance as there are a few updates on your most recent manifests for the Pulsar-manager I would like to leverage. Unfortunately, the release vote hasn't completed. https://lists.apache.org/thread/t4vlvk0nwd3gzo50t466cqy5h95l81gb . I'll follow up on the release voting thread. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Plugins support [pulsar-helm-chart]
lhotari commented on issue #491: URL: https://github.com/apache/pulsar-helm-chart/issues/491#issuecomment-203744 @aqeelat Contributions are welcome! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][offload] Fix NPE issue when read offload data [pulsar]
shibd closed pull request #21269: [fix][offload] Fix NPE issue when read offload data URL: https://github.com/apache/pulsar/pull/21269 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch branch-3.0 updated: [improve][offload] Apply autoSkipNonRecoverableData configuration to tiered storage (#22531)
This is an automated email from the ASF dual-hosted git repository.
baodi pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new ff8d3b73437 [improve][offload] Apply autoSkipNonRecoverableData
configuration to tiered storage (#22531)
ff8d3b73437 is described below
commit ff8d3b7343781fbb874d55635df27dd13f48b68e
Author: Baodi Shi
AuthorDate: Fri Apr 19 21:51:10 2024 +0800
[improve][offload] Apply autoSkipNonRecoverableData configuration to tiered
storage (#22531)
(cherry picked from commit fbf4cb71a3f3ed08786205dc5e60b810f3d62605)
---
.../impl/BlobStoreBackedInputStreamImpl.java | 8
.../jcloud/impl/BlobStoreBackedReadHandleImpl.java | 13 +-
.../impl/BlobStoreBackedReadHandleImplV2.java | 13 +-
.../jcloud/BlobStoreBackedInputStreamTest.java | 5 +-
...obStoreManagedLedgerOffloaderStreamingTest.java | 54 ++
.../impl/BlobStoreManagedLedgerOffloaderTest.java | 23 +
6 files changed, 110 insertions(+), 6 deletions(-)
diff --git
a/tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedInputStreamImpl.java
b/tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedInputStreamImpl.java
index 0dea46726f5..6cb60e14984 100644
---
a/tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedInputStreamImpl.java
+++
b/tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedInputStreamImpl.java
@@ -28,6 +28,7 @@ import
org.apache.bookkeeper.mledger.offload.jcloud.impl.DataBlockUtils.VersionC
import org.apache.pulsar.common.allocator.PulsarByteBufAllocator;
import org.apache.pulsar.common.naming.TopicName;
import org.jclouds.blobstore.BlobStore;
+import org.jclouds.blobstore.KeyNotFoundException;
import org.jclouds.blobstore.domain.Blob;
import org.jclouds.blobstore.options.GetOptions;
import org.slf4j.Logger;
@@ -95,6 +96,9 @@ public class BlobStoreBackedInputStreamImpl extends
BackedInputStream {
try {
long startReadTime = System.nanoTime();
Blob blob = blobStore.getBlob(bucket, key, new
GetOptions().range(startRange, endRange));
+if (blob == null) {
+throw new KeyNotFoundException(bucket, key, "");
+}
versionCheck.check(key, blob);
try (InputStream stream = blob.getPayload().openStream()) {
@@ -121,6 +125,10 @@ public class BlobStoreBackedInputStreamImpl extends
BackedInputStream {
if (null != this.offloaderStats) {
this.offloaderStats.recordReadOffloadError(this.topicName);
}
+// If the blob is not found, the original exception is thrown
and handled by the caller.
+if (e instanceof KeyNotFoundException) {
+throw e;
+}
throw new IOException("Error reading from BlobStore", e);
}
}
diff --git
a/tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedReadHandleImpl.java
b/tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedReadHandleImpl.java
index 5346be6a044..4f68f90370e 100644
---
a/tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedReadHandleImpl.java
+++
b/tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedReadHandleImpl.java
@@ -49,6 +49,7 @@ import
org.apache.bookkeeper.mledger.offload.jcloud.impl.DataBlockUtils.VersionC
import org.apache.pulsar.common.allocator.PulsarByteBufAllocator;
import org.apache.pulsar.common.naming.TopicName;
import org.jclouds.blobstore.BlobStore;
+import org.jclouds.blobstore.KeyNotFoundException;
import org.jclouds.blobstore.domain.Blob;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -202,7 +203,11 @@ public class BlobStoreBackedReadHandleImpl implements
ReadHandle {
} catch (Throwable t) {
log.error("Failed to read entries {} - {} from the offloader
in ledger {}",
firstEntry, lastEntry, ledgerId, t);
-promise.completeExceptionally(t);
+if (t instanceof KeyNotFoundException) {
+promise.completeExceptionally(new
BKException.BKNoSuchLedgerExistsException());
+} else {
+promise.completeExceptionally(t);
+}
entries.forEach(LedgerEntry::close);
}
});
@@ -265,7 +270,7 @@ public class BlobStoreBackedReadHandleImpl implements
ReadHandle {
VersionCheck
(pulsar) branch branch-3.1 updated: [improve][offload] Apply autoSkipNonRecoverableData configuration to tiered storage (#22531)
This is an automated email from the ASF dual-hosted git repository.
baodi pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new 88243208ba7 [improve][offload] Apply autoSkipNonRecoverableData
configuration to tiered storage (#22531)
88243208ba7 is described below
commit 88243208ba745f272d704dde7282b8ddc63fc30f
Author: Baodi Shi
AuthorDate: Fri Apr 19 21:51:10 2024 +0800
[improve][offload] Apply autoSkipNonRecoverableData configuration to tiered
storage (#22531)
(cherry picked from commit fbf4cb71a3f3ed08786205dc5e60b810f3d62605)
---
.../impl/BlobStoreBackedInputStreamImpl.java | 8
.../jcloud/impl/BlobStoreBackedReadHandleImpl.java | 13 +-
.../impl/BlobStoreBackedReadHandleImplV2.java | 13 +-
.../jcloud/BlobStoreBackedInputStreamTest.java | 5 +-
...obStoreManagedLedgerOffloaderStreamingTest.java | 54 ++
.../impl/BlobStoreManagedLedgerOffloaderTest.java | 23 +
6 files changed, 110 insertions(+), 6 deletions(-)
diff --git
a/tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedInputStreamImpl.java
b/tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedInputStreamImpl.java
index 0dea46726f5..6cb60e14984 100644
---
a/tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedInputStreamImpl.java
+++
b/tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedInputStreamImpl.java
@@ -28,6 +28,7 @@ import
org.apache.bookkeeper.mledger.offload.jcloud.impl.DataBlockUtils.VersionC
import org.apache.pulsar.common.allocator.PulsarByteBufAllocator;
import org.apache.pulsar.common.naming.TopicName;
import org.jclouds.blobstore.BlobStore;
+import org.jclouds.blobstore.KeyNotFoundException;
import org.jclouds.blobstore.domain.Blob;
import org.jclouds.blobstore.options.GetOptions;
import org.slf4j.Logger;
@@ -95,6 +96,9 @@ public class BlobStoreBackedInputStreamImpl extends
BackedInputStream {
try {
long startReadTime = System.nanoTime();
Blob blob = blobStore.getBlob(bucket, key, new
GetOptions().range(startRange, endRange));
+if (blob == null) {
+throw new KeyNotFoundException(bucket, key, "");
+}
versionCheck.check(key, blob);
try (InputStream stream = blob.getPayload().openStream()) {
@@ -121,6 +125,10 @@ public class BlobStoreBackedInputStreamImpl extends
BackedInputStream {
if (null != this.offloaderStats) {
this.offloaderStats.recordReadOffloadError(this.topicName);
}
+// If the blob is not found, the original exception is thrown
and handled by the caller.
+if (e instanceof KeyNotFoundException) {
+throw e;
+}
throw new IOException("Error reading from BlobStore", e);
}
}
diff --git
a/tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedReadHandleImpl.java
b/tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedReadHandleImpl.java
index 5346be6a044..4f68f90370e 100644
---
a/tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedReadHandleImpl.java
+++
b/tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedReadHandleImpl.java
@@ -49,6 +49,7 @@ import
org.apache.bookkeeper.mledger.offload.jcloud.impl.DataBlockUtils.VersionC
import org.apache.pulsar.common.allocator.PulsarByteBufAllocator;
import org.apache.pulsar.common.naming.TopicName;
import org.jclouds.blobstore.BlobStore;
+import org.jclouds.blobstore.KeyNotFoundException;
import org.jclouds.blobstore.domain.Blob;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -202,7 +203,11 @@ public class BlobStoreBackedReadHandleImpl implements
ReadHandle {
} catch (Throwable t) {
log.error("Failed to read entries {} - {} from the offloader
in ledger {}",
firstEntry, lastEntry, ledgerId, t);
-promise.completeExceptionally(t);
+if (t instanceof KeyNotFoundException) {
+promise.completeExceptionally(new
BKException.BKNoSuchLedgerExistsException());
+} else {
+promise.completeExceptionally(t);
+}
entries.forEach(LedgerEntry::close);
}
});
@@ -265,7 +270,7 @@ public class BlobStoreBackedReadHandleImpl implements
ReadHandle {
VersionCheck
(pulsar) branch branch-3.2 updated: [improve][offload] Apply autoSkipNonRecoverableData configuration to tiered storage (#22531)
This is an automated email from the ASF dual-hosted git repository.
baodi pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 14841768466 [improve][offload] Apply autoSkipNonRecoverableData
configuration to tiered storage (#22531)
14841768466 is described below
commit 1484176846674b032f6745b6e8537139226994a2
Author: Baodi Shi
AuthorDate: Fri Apr 19 21:51:10 2024 +0800
[improve][offload] Apply autoSkipNonRecoverableData configuration to tiered
storage (#22531)
(cherry picked from commit fbf4cb71a3f3ed08786205dc5e60b810f3d62605)
---
.../impl/BlobStoreBackedInputStreamImpl.java | 8
.../jcloud/impl/BlobStoreBackedReadHandleImpl.java | 13 +-
.../impl/BlobStoreBackedReadHandleImplV2.java | 13 +-
.../jcloud/BlobStoreBackedInputStreamTest.java | 5 +-
...obStoreManagedLedgerOffloaderStreamingTest.java | 54 ++
.../impl/BlobStoreManagedLedgerOffloaderTest.java | 23 +
6 files changed, 110 insertions(+), 6 deletions(-)
diff --git
a/tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedInputStreamImpl.java
b/tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedInputStreamImpl.java
index 0dea46726f5..6cb60e14984 100644
---
a/tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedInputStreamImpl.java
+++
b/tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedInputStreamImpl.java
@@ -28,6 +28,7 @@ import
org.apache.bookkeeper.mledger.offload.jcloud.impl.DataBlockUtils.VersionC
import org.apache.pulsar.common.allocator.PulsarByteBufAllocator;
import org.apache.pulsar.common.naming.TopicName;
import org.jclouds.blobstore.BlobStore;
+import org.jclouds.blobstore.KeyNotFoundException;
import org.jclouds.blobstore.domain.Blob;
import org.jclouds.blobstore.options.GetOptions;
import org.slf4j.Logger;
@@ -95,6 +96,9 @@ public class BlobStoreBackedInputStreamImpl extends
BackedInputStream {
try {
long startReadTime = System.nanoTime();
Blob blob = blobStore.getBlob(bucket, key, new
GetOptions().range(startRange, endRange));
+if (blob == null) {
+throw new KeyNotFoundException(bucket, key, "");
+}
versionCheck.check(key, blob);
try (InputStream stream = blob.getPayload().openStream()) {
@@ -121,6 +125,10 @@ public class BlobStoreBackedInputStreamImpl extends
BackedInputStream {
if (null != this.offloaderStats) {
this.offloaderStats.recordReadOffloadError(this.topicName);
}
+// If the blob is not found, the original exception is thrown
and handled by the caller.
+if (e instanceof KeyNotFoundException) {
+throw e;
+}
throw new IOException("Error reading from BlobStore", e);
}
}
diff --git
a/tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedReadHandleImpl.java
b/tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedReadHandleImpl.java
index 5346be6a044..4f68f90370e 100644
---
a/tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedReadHandleImpl.java
+++
b/tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedReadHandleImpl.java
@@ -49,6 +49,7 @@ import
org.apache.bookkeeper.mledger.offload.jcloud.impl.DataBlockUtils.VersionC
import org.apache.pulsar.common.allocator.PulsarByteBufAllocator;
import org.apache.pulsar.common.naming.TopicName;
import org.jclouds.blobstore.BlobStore;
+import org.jclouds.blobstore.KeyNotFoundException;
import org.jclouds.blobstore.domain.Blob;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -202,7 +203,11 @@ public class BlobStoreBackedReadHandleImpl implements
ReadHandle {
} catch (Throwable t) {
log.error("Failed to read entries {} - {} from the offloader
in ledger {}",
firstEntry, lastEntry, ledgerId, t);
-promise.completeExceptionally(t);
+if (t instanceof KeyNotFoundException) {
+promise.completeExceptionally(new
BKException.BKNoSuchLedgerExistsException());
+} else {
+promise.completeExceptionally(t);
+}
entries.forEach(LedgerEntry::close);
}
});
@@ -265,7 +270,7 @@ public class BlobStoreBackedReadHandleImpl implements
ReadHandle {
VersionCheck
(pulsar) branch master updated (2badcf6bd0b -> fbf4cb71a3f)
This is an automated email from the ASF dual-hosted git repository. baodi pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git from 2badcf6bd0b [fix][broker] Fix NPE causing dispatching to stop when using Key_Shared mode and allowOutOfOrderDelivery=true (#22533) add fbf4cb71a3f [improve][offload] Apply autoSkipNonRecoverableData configuration to tiered storage (#22531) No new revisions were added by this update. Summary of changes: .../impl/BlobStoreBackedInputStreamImpl.java | 8 .../jcloud/impl/BlobStoreBackedReadHandleImpl.java | 13 +- .../impl/BlobStoreBackedReadHandleImplV2.java | 13 +- .../jcloud/BlobStoreBackedInputStreamTest.java | 5 +- ...obStoreManagedLedgerOffloaderStreamingTest.java | 54 ++ .../impl/BlobStoreManagedLedgerOffloaderTest.java | 23 + 6 files changed, 110 insertions(+), 6 deletions(-)
Re: [PR] [improve][offload] Apply autoSkipNonRecoverableData configuration to tiered storage [pulsar]
shibd merged PR #22531: URL: https://github.com/apache/pulsar/pull/22531 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] Plugins support [pulsar-helm-chart]
aqeelat opened a new issue, #491: URL: https://github.com/apache/pulsar-helm-chart/issues/491 **Is your feature request related to a problem? Please describe.** We're currently using streamnative protocol plugins and it would be great if we can just set a list urls or local files in the chart and then have them added in the relevant pods (or all). **Describe the solution you'd like** Possible solutions: 1. adding a top level section in values.yaml called `plugins` and then we can add the stuff we want to it 2. allowing us to declare custom initContainers (#240) **Describe alternatives you've considered** A clear and concise description of any alternative solutions or features you've considered. 1. custom images (was an overkill) 2. using volumeMounts. This a good option but it means we have to download the plugins and prepare the mounted directory ourselves. I'm not sure how this will play with argo but I didn't test it. 3. using converting the chart into a Kustomize and then adding the initContainers. (Yet to be tested) [approach 1](https://github.com/kubernetes-sigs/kustomize/blob/master/examples/chart.md) or probably [approach 2](https://github.com/thomastaylor312/advanced-helm-demos/tree/master/post-render) **Additional context** This will be a quality of life improvement for us. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] Cannot build on Apple Arm [pulsar]
lhotari commented on issue #22538: URL: https://github.com/apache/pulsar/issues/22538#issuecomment-2066595291 > 3.1.2, or whatever latest @chenrui333 It seems that the Pulsar version in homebrew is outdated. Should use Pulsar version 3.2.2. Do you have a chance to upgrade to 3.2.2 and retry? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] chore(deps): bump golang.org/x/net from 0.17.0 to 0.23.0 [pulsar-client-go]
dependabot[bot] opened a new pull request, #1209: URL: https://github.com/apache/pulsar-client-go/pull/1209 Bumps [golang.org/x/net](https://github.com/golang/net) from 0.17.0 to 0.23.0. Commits https://github.com/golang/net/commit/c48da131589f122489348be5dfbcb6457640046f;>c48da13 http2: fix TestServerContinuationFlood flakes https://github.com/golang/net/commit/762b58d1cf6e0779780decad89c6c1523386638d;>762b58d http2: fix tipos in comment https://github.com/golang/net/commit/ba872109ef2dc8f1da778651bd1fd3792d0e4587;>ba87210 http2: close connections when receiving too many headers https://github.com/golang/net/commit/ebc8168ac8ac742194df729305175940790c55a2;>ebc8168 all: fix some typos https://github.com/golang/net/commit/3678185f8a652e52864c44049a9ea96b7bcc066a;>3678185 http2: make TestCanonicalHeaderCacheGrowth faster https://github.com/golang/net/commit/448c44f9287b6745f958d74aa2a17ec7761c2f13;>448c44f http2: remove clientTester https://github.com/golang/net/commit/c7877ac4213b2f859831366f5a35b353e0dc9f66;>c7877ac http2: convert the remaining clientTester tests to testClientConn https://github.com/golang/net/commit/d8870b0bf2f2426fc8d19a9332f652da5c25418f;>d8870b0 http2: use synthetic time in TestIdleConnTimeout https://github.com/golang/net/commit/d73acffdc9493532acb85777105bb4a351eea702;>d73acff http2: only set up deadline when Server.IdleTimeout is positive https://github.com/golang/net/commit/89f602b7bbf237abe0467031a18b42fc742ced08;>89f602b http2: validate client/outgoing trailers Additional commits viewable in https://github.com/golang/net/compare/v0.17.0...v0.23.0;>compare view [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/pulsar-client-go/network/alerts). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar-client-go) branch dependabot/go_modules/golang.org/x/net-0.23.0 created (now 43818365)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/golang.org/x/net-0.23.0 in repository https://gitbox.apache.org/repos/asf/pulsar-client-go.git at 43818365 chore(deps): bump golang.org/x/net from 0.17.0 to 0.23.0 No new revisions were added by this update.
Re: [PR] Bump golang.org/x/net from 0.17.0 to 0.23.0 in /pulsar-function-go [pulsar]
github-actions[bot] commented on PR #22540: URL: https://github.com/apache/pulsar/pull/22540#issuecomment-2066475264 @dependabot[bot] Please add the following content to your PR description and select a checkbox: ``` - [ ] `doc` - [ ] `doc-required` - [ ] `doc-not-needed` - [ ] `doc-complete` ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch dependabot/go_modules/pulsar-function-go/golang.org/x/net-0.23.0 created (now e29291a9a36)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/pulsar-function-go/golang.org/x/net-0.23.0 in repository https://gitbox.apache.org/repos/asf/pulsar.git at e29291a9a36 Bump golang.org/x/net from 0.17.0 to 0.23.0 in /pulsar-function-go No new revisions were added by this update.
[PR] Bump golang.org/x/net from 0.17.0 to 0.23.0 in /pulsar-function-go [pulsar]
dependabot[bot] opened a new pull request, #22540: URL: https://github.com/apache/pulsar/pull/22540 Bumps [golang.org/x/net](https://github.com/golang/net) from 0.17.0 to 0.23.0. Commits https://github.com/golang/net/commit/c48da131589f122489348be5dfbcb6457640046f;>c48da13 http2: fix TestServerContinuationFlood flakes https://github.com/golang/net/commit/762b58d1cf6e0779780decad89c6c1523386638d;>762b58d http2: fix tipos in comment https://github.com/golang/net/commit/ba872109ef2dc8f1da778651bd1fd3792d0e4587;>ba87210 http2: close connections when receiving too many headers https://github.com/golang/net/commit/ebc8168ac8ac742194df729305175940790c55a2;>ebc8168 all: fix some typos https://github.com/golang/net/commit/3678185f8a652e52864c44049a9ea96b7bcc066a;>3678185 http2: make TestCanonicalHeaderCacheGrowth faster https://github.com/golang/net/commit/448c44f9287b6745f958d74aa2a17ec7761c2f13;>448c44f http2: remove clientTester https://github.com/golang/net/commit/c7877ac4213b2f859831366f5a35b353e0dc9f66;>c7877ac http2: convert the remaining clientTester tests to testClientConn https://github.com/golang/net/commit/d8870b0bf2f2426fc8d19a9332f652da5c25418f;>d8870b0 http2: use synthetic time in TestIdleConnTimeout https://github.com/golang/net/commit/d73acffdc9493532acb85777105bb4a351eea702;>d73acff http2: only set up deadline when Server.IdleTimeout is positive https://github.com/golang/net/commit/89f602b7bbf237abe0467031a18b42fc742ced08;>89f602b http2: validate client/outgoing trailers Additional commits viewable in https://github.com/golang/net/compare/v0.17.0...v0.23.0;>compare view [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/pulsar/network/alerts). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Bump golang.org/x/net from 0.17.0 to 0.23.0 in /pulsar-function-go/examples [pulsar]
github-actions[bot] commented on PR #22539: URL: https://github.com/apache/pulsar/pull/22539#issuecomment-2066473125 @dependabot[bot] Please add the following content to your PR description and select a checkbox: ``` - [ ] `doc` - [ ] `doc-required` - [ ] `doc-not-needed` - [ ] `doc-complete` ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch dependabot/go_modules/pulsar-function-go/examples/golang.org/x/net-0.23.0 created (now 2c5fd167005)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/pulsar-function-go/examples/golang.org/x/net-0.23.0 in repository https://gitbox.apache.org/repos/asf/pulsar.git at 2c5fd167005 Bump golang.org/x/net in /pulsar-function-go/examples No new revisions were added by this update.
[PR] Bump golang.org/x/net from 0.17.0 to 0.23.0 in /pulsar-function-go/examples [pulsar]
dependabot[bot] opened a new pull request, #22539: URL: https://github.com/apache/pulsar/pull/22539 Bumps [golang.org/x/net](https://github.com/golang/net) from 0.17.0 to 0.23.0. Commits https://github.com/golang/net/commit/c48da131589f122489348be5dfbcb6457640046f;>c48da13 http2: fix TestServerContinuationFlood flakes https://github.com/golang/net/commit/762b58d1cf6e0779780decad89c6c1523386638d;>762b58d http2: fix tipos in comment https://github.com/golang/net/commit/ba872109ef2dc8f1da778651bd1fd3792d0e4587;>ba87210 http2: close connections when receiving too many headers https://github.com/golang/net/commit/ebc8168ac8ac742194df729305175940790c55a2;>ebc8168 all: fix some typos https://github.com/golang/net/commit/3678185f8a652e52864c44049a9ea96b7bcc066a;>3678185 http2: make TestCanonicalHeaderCacheGrowth faster https://github.com/golang/net/commit/448c44f9287b6745f958d74aa2a17ec7761c2f13;>448c44f http2: remove clientTester https://github.com/golang/net/commit/c7877ac4213b2f859831366f5a35b353e0dc9f66;>c7877ac http2: convert the remaining clientTester tests to testClientConn https://github.com/golang/net/commit/d8870b0bf2f2426fc8d19a9332f652da5c25418f;>d8870b0 http2: use synthetic time in TestIdleConnTimeout https://github.com/golang/net/commit/d73acffdc9493532acb85777105bb4a351eea702;>d73acff http2: only set up deadline when Server.IdleTimeout is positive https://github.com/golang/net/commit/89f602b7bbf237abe0467031a18b42fc742ced08;>89f602b http2: validate client/outgoing trailers Additional commits viewable in https://github.com/golang/net/compare/v0.17.0...v0.23.0;>compare view [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/pulsar/network/alerts). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] [Bug] Cannot build on Apple Arm [pulsar]
chenrui333 opened a new issue, #22538: URL: https://github.com/apache/pulsar/issues/22538 ### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Read release policy - [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker. ### Version - 3.1.2, or whatever latest ### Minimal reproduce step - `brew install --build-from-source apache-pulsar` would show the build issue ### What did you expect to see? build pass ### What did you see instead? ``` Caused by: org.apache.maven.plugin.MojoExecutionException: An error occurred while invoking protoc: Error while executing process. ... Caused by: org.codehaus.plexus.util.cli.CommandLineException: Error while executing process. ... Caused by: java.io.IOException: Cannot run program "/private/tmp/apache-pulsar-20240411-8261-zm4by3/apache-pulsar-3.1.2-src/managed-ledger/target/protoc-plugins/protoc-3.19.6-osx-aarch_64.exe": error=86, Bad CPU type in executable ... Caused by: java.io.IOException: error=86, Bad CPU type in executable ... ``` ### Anything else? full build log in here, https://github.com/Homebrew/homebrew-core/actions/runs/8655513598/job/23734842487?pr=168487 relates to: - https://github.com/apache/pulsar/issues/16639 - https://github.com/Homebrew/homebrew-core/pull/168487 ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Consumer able to receive message which is not matching the regex pattern [pulsar]
ragaur-tibco commented on issue #22529: URL: https://github.com/apache/pulsar/issues/22529#issuecomment-2066312881 ok Thank you @visortelle But for non-persistent I was not able to do multi topic subscription and as you said "Once in about ~5 runs I see some messages from non-persistent topic" but for me it is not working -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Topic creation not working properly [pulsar]
ragaur-tibco commented on issue #22527: URL: https://github.com/apache/pulsar/issues/22527#issuecomment-2066184414 but it is not working properly with non-persistent topics @visortelle -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [improve] [broker] Create partitioned topics automatically when enable topic level replication [pulsar]
Demogorgon314 commented on code in PR #22537:
URL: https://github.com/apache/pulsar/pull/22537#discussion_r1572038896
##
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/OneWayReplicatorTest.java:
##
@@ -153,4 +172,90 @@ public void
testTopicCloseWhenInternalProducerCloseErrorOnce() throws Exception
admin2.topics().delete(topicName);
});
}
+
+@Test
+public void testPartitionedTopicLevelReplication() throws Exception {
+final String topicName = BrokerTestUtil.newUniqueName("persistent://"
+ nonReplicatedNamespace + "/tp_");
+final String partition0 =
TopicName.get(topicName).getPartition(0).toString();
+final String partition1 =
TopicName.get(topicName).getPartition(1).toString();
+admin1.topics().createPartitionedTopic(topicName, 2);
+admin1.topics().setReplicationClusters(topicName,
Arrays.asList(cluster1, cluster2));
+// Check the partitioned topic has been created at the remote cluster.
+PartitionedTopicMetadata topicMetadata2 =
admin2.topics().getPartitionedTopicMetadata(topicName);
+assertEquals(topicMetadata2.partitions, 2);
+// cleanup.
+admin1.topics().setReplicationClusters(topicName,
Arrays.asList(cluster1));
+waitReplicatorStopped(partition0);
+waitReplicatorStopped(partition1);
+admin1.topics().deletePartitionedTopic(topicName);
+admin2.topics().deletePartitionedTopic(topicName);
+}
+
+@Test
+public void testPartitionedTopicLevelReplicationRemoteTopicExist() throws
Exception {
+final String topicName = BrokerTestUtil.newUniqueName("persistent://"
+ nonReplicatedNamespace + "/tp_");
+final String partition0 =
TopicName.get(topicName).getPartition(0).toString();
+final String partition1 =
TopicName.get(topicName).getPartition(1).toString();
+admin1.topics().createPartitionedTopic(topicName, 2);
+admin2.topics().createPartitionedTopic(topicName, 2);
+admin1.topics().setReplicationClusters(topicName,
Arrays.asList(cluster1, cluster2));
+// Check the partitioned topic has been created at the remote cluster.
+PartitionedTopicMetadata topicMetadata2 =
admin2.topics().getPartitionedTopicMetadata(topicName);
+assertEquals(topicMetadata2.partitions, 2);
+// cleanup.
+admin1.topics().setReplicationClusters(topicName,
Arrays.asList(cluster1));
+waitReplicatorStopped(partition0);
+waitReplicatorStopped(partition1);
+admin1.topics().deletePartitionedTopic(topicName);
+admin2.topics().deletePartitionedTopic(topicName);
+}
+
+@Test
+public void testPartitionedTopicLevelReplicationRemoteConflictTopicExist()
throws Exception {
Review Comment:
When the remote cluster has 1 partition for the topic, and then we try to
create the topic with 2 partition on the source cluster, should we update the
remote cluster's topic partition to 2?
##
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/OneWayReplicatorTest.java:
##
@@ -153,4 +172,90 @@ public void
testTopicCloseWhenInternalProducerCloseErrorOnce() throws Exception
admin2.topics().delete(topicName);
});
}
+
+@Test
+public void testPartitionedTopicLevelReplication() throws Exception {
+final String topicName = BrokerTestUtil.newUniqueName("persistent://"
+ nonReplicatedNamespace + "/tp_");
+final String partition0 =
TopicName.get(topicName).getPartition(0).toString();
+final String partition1 =
TopicName.get(topicName).getPartition(1).toString();
+admin1.topics().createPartitionedTopic(topicName, 2);
+admin1.topics().setReplicationClusters(topicName,
Arrays.asList(cluster1, cluster2));
+// Check the partitioned topic has been created at the remote cluster.
+PartitionedTopicMetadata topicMetadata2 =
admin2.topics().getPartitionedTopicMetadata(topicName);
+assertEquals(topicMetadata2.partitions, 2);
+// cleanup.
+admin1.topics().setReplicationClusters(topicName,
Arrays.asList(cluster1));
+waitReplicatorStopped(partition0);
+waitReplicatorStopped(partition1);
+admin1.topics().deletePartitionedTopic(topicName);
+admin2.topics().deletePartitionedTopic(topicName);
+}
+
+@Test
+public void testPartitionedTopicLevelReplicationRemoteTopicExist() throws
Exception {
+final String topicName = BrokerTestUtil.newUniqueName("persistent://"
+ nonReplicatedNamespace + "/tp_");
+final String partition0 =
TopicName.get(topicName).getPartition(0).toString();
+final String partition1 =
TopicName.get(topicName).getPartition(1).toString();
+admin1.topics().createPartitionedTopic(topicName, 2);
+admin2.topics().createPartitionedTopic(topicName, 2);
+admin1.topics().setReplicationClusters(topicName,
svn commit: r68647 - in /dev/pulsar/pulsar-dotpulsar-3.2.1-rc.1: pulsar-dotpulsar-3.2.1-src.tar.gz pulsar-dotpulsar-3.2.1-src.tar.gz.asc pulsar-dotpulsar-3.2.1-src.tar.gz.sha512
Author: djensen Date: Fri Apr 19 08:43:06 2024 New Revision: 68647 Log: Staging artifacts and signature for DotPulsar 3.2.1-rc.1 Added: dev/pulsar/pulsar-dotpulsar-3.2.1-rc.1/pulsar-dotpulsar-3.2.1-src.tar.gz (with props) dev/pulsar/pulsar-dotpulsar-3.2.1-rc.1/pulsar-dotpulsar-3.2.1-src.tar.gz.asc dev/pulsar/pulsar-dotpulsar-3.2.1-rc.1/pulsar-dotpulsar-3.2.1-src.tar.gz.sha512 Added: dev/pulsar/pulsar-dotpulsar-3.2.1-rc.1/pulsar-dotpulsar-3.2.1-src.tar.gz == Binary file - no diff available. Propchange: dev/pulsar/pulsar-dotpulsar-3.2.1-rc.1/pulsar-dotpulsar-3.2.1-src.tar.gz -- svn:mime-type = application/octet-stream Added: dev/pulsar/pulsar-dotpulsar-3.2.1-rc.1/pulsar-dotpulsar-3.2.1-src.tar.gz.asc == --- dev/pulsar/pulsar-dotpulsar-3.2.1-rc.1/pulsar-dotpulsar-3.2.1-src.tar.gz.asc (added) +++ dev/pulsar/pulsar-dotpulsar-3.2.1-rc.1/pulsar-dotpulsar-3.2.1-src.tar.gz.asc Fri Apr 19 08:43:06 2024 @@ -0,0 +1,16 @@ +-BEGIN PGP SIGNATURE- + +iQIzBAABCgAdFiEEBNpmyfPM8T5xTvymkG5uByPUwhUFAmYiLpUACgkQkG5uByPU +whXghRAAsk/b1f5NJGkJBKIe3o9Vgy3EVeUFBE/YyVUeQvJsk+pfTfPtR0Dwj4pr +khYFvMhWC48rRdEEshef8hjyRY5upoKMUFu93L3febJxQ/75lObmtYd5ZCtGrSp3 +JNUhQ4d/jAt0bt5SQ4pn8GX2Y6bTXT5Wcrz+sNWBXvo6ZLxrdAcAlV5NwVyMc4kz +3qWrwtY79/tQhKj9dPIeKdXbhR+A45IcgOLqNttrDvLHhA40bAamdCeOuV22/end +MjwPXrQ2Vuqb+4lKoHt10TYW0tbm5YtPGVt0VfSgscIxllEo4BaWwGfqHEUb0fZN +ybqye+WKLwUV94LFQ/0oT91fqqfM0AvZ9V9r0xbHamH0+J67EEv1JCIO8wWRYEJk +XQTRyjtMr/DyOMKMPagFTTjHzxm0ImS3TrEJHLqXh+3/fKiQsenAhhadV407ZLsV +KH/nwnvq0ElHyepfUuMOroCtm0YMJmOHI+37MjK5gpmRjBmFo+CEEr0b2zuJhBUk +zpftqdfb1v8DTN+GL+yFqJBem3Zu1NpJxUzDjmSAoOhHPxluhDcLJTmurjJVJ4ui +v2sKe1l2fWzmbLMi+XNp4k4s1Ir7YwfCDs37RUO3NUvUk8yg1kj6njQG17CVfEJ+ +Ehi/vXIdr5QI8BZgkElb/Yecfus6EZOyU7H4qN9h5c+4fNL9Y0A= +=f6pk +-END PGP SIGNATURE- Added: dev/pulsar/pulsar-dotpulsar-3.2.1-rc.1/pulsar-dotpulsar-3.2.1-src.tar.gz.sha512 == --- dev/pulsar/pulsar-dotpulsar-3.2.1-rc.1/pulsar-dotpulsar-3.2.1-src.tar.gz.sha512 (added) +++ dev/pulsar/pulsar-dotpulsar-3.2.1-rc.1/pulsar-dotpulsar-3.2.1-src.tar.gz.sha512 Fri Apr 19 08:43:06 2024 @@ -0,0 +1 @@ +7a3526a27db3cae839873d60ff4823dc9225c758b8551965274cb068a1982ead73c9b0d4eea0d733d0980c606b4476f2c4d1b37054ee2d07efd14f3c1e9647ea pulsar-dotpulsar-3.2.1-src.tar.gz
svn commit: r68646 - /dev/pulsar/pulsar-dotpulsar-3.2.1-rc.1/
Author: djensen Date: Fri Apr 19 08:43:02 2024 New Revision: 68646 Log: Create DotPulsar pre-release dir Added: dev/pulsar/pulsar-dotpulsar-3.2.1-rc.1/
Re: [PR] [fix][broker] Support lookup options for extensible load manager [pulsar]
BewareMyPower commented on code in PR #22487:
URL: https://github.com/apache/pulsar/pull/22487#discussion_r1572038446
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/LoadManager.java:
##
@@ -63,7 +65,7 @@ public interface LoadManager {
Optional getLeastLoaded(ServiceUnitId su) throws Exception;
default CompletableFuture> findBrokerServiceUrl(
-Optional topic, ServiceUnitId bundle) {
+Optional topic, NamespaceBundle bundle,
LookupOptions options) {
throw new UnsupportedOperationException();
}
Review Comment:
It's better to keep the API compatibility since it's an interface that can
be configured. You can keep the 2nd parameter as `ServiceUnitId` and add a new
default method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [I] Consumer able to receive message which is not matching the regex pattern [pulsar]
visortelle commented on issue #22529: URL: https://github.com/apache/pulsar/issues/22529#issuecomment-2066092324 Related: https://github.com/apache/pulsar/issues/19798 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar-dotpulsar) tag 3.2.1-rc.1 created (now ffa5665)
This is an automated email from the ASF dual-hosted git repository. djensen pushed a change to tag 3.2.1-rc.1 in repository https://gitbox.apache.org/repos/asf/pulsar-dotpulsar.git at ffa5665 (commit) No new revisions were added by this update.
Re: [I] Topic creation not working properly [pulsar]
visortelle commented on issue #22527:
URL: https://github.com/apache/pulsar/issues/22527#issuecomment-2066042578
It may be hard to catch, but for some reason, it sometimes works. This time
I ran 20+ times to get messages from the non-persistent `topic-c`.
Added logs for subscriptions per topic.
Code
```scala
pulsarAdmin.topics.getList("new-tenant/new-namespace").asScala
.foreach(pulsarAdmin.topics.delete(_))
val consumer = pulsarClient.newConsumer()
.topicsPattern("new-tenant/new-namespace/.*".r.pattern)
.subscriptionName("new-subscription")
.patternAutoDiscoveryPeriod(1, TimeUnit.SECONDS)
.subscriptionTopicsMode(RegexSubscriptionMode.AllTopics)
.messageListener(new MessageListener[Array[Byte]] {
override def received(consumer:
org.apache.pulsar.client.api.Consumer[Array[Byte]], msg:
org.apache.pulsar.client.api.Message[Array[Byte]]): Unit = {
println(s"Received: ${msg.getValue.mkString(",")}. From topic:
${msg.getTopicName}")
consumer.acknowledge(msg)
}
})
.subscribe()
val topics = Vector(
s"persistent://new-tenant/new-namespace/topic-a",
s"persistent://new-tenant/new-namespace/topic-b",
s"non-persistent://new-tenant/new-namespace/topic-c",
s"non-persistent://new-tenant/new-namespace/topic-d",
)
topics.foreach(topic =>
val producer = pulsarClient.newProducer.topic(topic).create()
val from = 4096L
val to = from + 10
for (i <- from until to)
producer.send(scala.math.BigInt(i).toByteArray)
producer.flush()
producer.close()
)
println(s"Sleep started at ${java.time.LocalTime.now()}")
Thread.sleep(10 * 1000)
println(s"Sleep finished at ${java.time.LocalTime.now()}")
// All the topics should be discovered at this moment
def logSubscriptions =
topics.foreach(topic =>
val subscriptions =
pulsarAdmin.topics.getSubscriptions(topic).asScala
println(s"Subscriptions for topic $topic: $subscriptions")
)
logSubscriptions
topics.foreach(topic =>
val producer = pulsarClient.newProducer.topic(topic).create()
val from = 4096L
val to = from + 10
for (i <- from until to)
producer.send(scala.math.BigInt(i).toByteArray)
producer.flush()
producer.close()
)
logSubscriptions
println(s"Sleep started at ${java.time.LocalTime.now()}")
Thread.sleep(10 * 5)
println(s"Sleep finished at ${java.time.LocalTime.now()}")
logSubscriptions
```
Logs
```
11:55:01.347 [pulsar-client-io-3-3] INFO
o.a.p.c.i.ProducerStatsRecorderImpl - Starting Pulsar producer perf with
config:
{"topicName":"persistent://new-tenant/new-namespace/topic-a","producerName":null,"sendTimeoutMs":3,"blockIfQueueFull":false,"maxPendingMessages":0,"maxPendingMessagesAcrossPartitions":0,"messageRoutingMode":"RoundRobinPartition","hashingScheme":"JavaStringHash","cryptoFailureAction":"FAIL","batchingMaxPublishDelayMicros":1000,"batchingPartitionSwitchFrequencyByPublishDelay":10,"batchingMaxMessages":1000,"batchingMaxBytes":131072,"batchingEnabled":true,"chunkingEnabled":false,"chunkMaxMessageSize":-1,"encryptionKeys":[],"compressionType":"NONE","initialSequenceId":null,"autoUpdatePartitions":true,"autoUpdatePartitionsIntervalSeconds":60,"multiSchema":true,"accessMode":"Shared","lazyStartPartitionedProducers":false,"properties":{},"initialSubscriptionName":null}
11:55:01.435 [pulsar-client-io-3-3] INFO
o.a.p.c.i.ProducerStatsRecorderImpl - Pulsar client config:
{"serviceUrl":"pulsar://localhost:6650","authPluginClassName":null,"authParams":null,"authParamMap":null,"operationTimeoutMs":3,"lookupTimeoutMs":3,"statsIntervalSeconds":60,"numIoThreads":12,"numListenerThreads":12,"connectionsPerBroker":1,"connectionMaxIdleSeconds":180,"useTcpNoDelay":true,"useTls":false,"tlsKeyFilePath":null,"tlsCertificateFilePath":null,"tlsTrustCertsFilePath":null,"tlsAllowInsecureConnection":false,"tlsHostnameVerificationEnable":false,"concurrentLookupRequest":5000,"maxLookupRequest":5,"maxLookupRedirects":20,"maxNumberOfRejectedRequestPerConnection":50,"keepAliveIntervalSeconds":30,"connectionTimeoutMs":1,"requestTimeoutMs":6,"readTimeoutMs":6,"autoCertRefreshSeconds":300,"initialBackoffIntervalNanos":1,"maxBackoffIntervalNanos":600,"enableBusyWait":false,"listenerName":null,"useKeyStoreTls":false,"sslProvider":null,"
(pulsar-dotpulsar) branch master updated: Make ready for release 3.2.1-rc.1
This is an automated email from the ASF dual-hosted git repository. djensen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar-dotpulsar.git The following commit(s) were added to refs/heads/master by this push: new ea4f136 Make ready for release 3.2.1-rc.1 ea4f136 is described below commit ea4f1360c991fbc52932db95398aebc69273f101 Author: entvex <1580435+ent...@users.noreply.github.com> AuthorDate: Fri Apr 19 10:02:14 2024 +0200 Make ready for release 3.2.1-rc.1 Make ready for release 3.2.1-rc.1 --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8761c97..b5c4b8c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [Unreleased] +## [3.2.1-rc.1] - 2024-04-09 ### Fixed
(pulsar-dotpulsar) branch master updated: Make ready for release 3.2.1-rc.1
This is an automated email from the ASF dual-hosted git repository. djensen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar-dotpulsar.git The following commit(s) were added to refs/heads/master by this push: new ffa5665 Make ready for release 3.2.1-rc.1 ffa5665 is described below commit ffa5665e0bfe701427e9b2dc0d06cde40292e6b5 Author: entvex <1580435+ent...@users.noreply.github.com> AuthorDate: Fri Apr 19 10:03:27 2024 +0200 Make ready for release 3.2.1-rc.1 --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b5c4b8c..d75ba78 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [3.2.1-rc.1] - 2024-04-09 +## [3.2.1-rc.1] - 2024-04-19 ### Fixed
Re: [I] Topic creation not working properly [pulsar]
ragaur-tibco commented on issue #22527: URL: https://github.com/apache/pulsar/issues/22527#issuecomment-2066004408 @visortelle I tried multiple times but I was not able to receive any messages from non-persistent topics even if I select `NonPersistentOnly` or `AllTopics` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Topic creation not working properly [pulsar]
visortelle commented on issue #22527:
URL: https://github.com/apache/pulsar/issues/22527#issuecomment-2065928960
@ragaur-tibco I'm also unable to receive messages from non-persistent topics
for some reason.
But it looks flaky. Once in about ~5 runs I see some messages from
non-persistent `topic-c`.
Code:
```scala
// Cleanup
pulsarAdmin.topics.getList("new-tenant/new-namespace").asScala
.foreach(pulsarAdmin.topics.delete(_))
val consumer = pulsarClient.newConsumer()
.topicsPattern("new-tenant/new-namespace/.*".r.pattern)
.subscriptionName("new-subscription")
.patternAutoDiscoveryPeriod(1, TimeUnit.SECONDS)
.subscriptionTopicsMode(RegexSubscriptionMode.AllTopics)
.messageListener(new MessageListener[Array[Byte]] {
override def received(consumer:
org.apache.pulsar.client.api.Consumer[Array[Byte]], msg:
org.apache.pulsar.client.api.Message[Array[Byte]]): Unit = {
println(s"Received: ${msg.getValue.mkString(",")}. From topic:
${msg.getTopicName}")
consumer.acknowledge(msg)
}
})
.subscribe()
val topics = Vector(
s"persistent://new-tenant/new-namespace/topic-a",
s"persistent://new-tenant/new-namespace/topic-b",
s"non-persistent://new-tenant/new-namespace/topic-c",
s"non-persistent://new-tenant/new-namespace/topic-d",
)
topics.foreach(topic =>
val producer = pulsarClient.newProducer.topic(topic).create()
val from = 4096L
val to = from + 10
for (i <- from until to)
producer.send(scala.math.BigInt(i).toByteArray)
producer.flush()
producer.close()
)
println(s"Sleep started at ${java.time.LocalTime.now()}")
Thread.sleep(10 * 1000)
println(s"Sleep finished at ${java.time.LocalTime.now()}")
// All the topics should be discovered at this moment
topics.foreach(topic =>
val producer = pulsarClient.newProducer.topic(topic).create()
val from = 4096L
val to = from + 10
for (i <- from until to)
producer.send(scala.math.BigInt(i).toByteArray)
producer.flush()
producer.close()
)
```
Logs:
```
11:18:06.573 [ZScheduler-Worker-4] INFO o.a.p.c.i.ProducerStatsRecorderImpl
- Starting Pulsar producer perf with config:
{"topicName":"persistent://new-tenant/new-namespace/topic-a","producerName":null,"sendTimeoutMs":3,"blockIfQueueFull":false,"maxPendingMessages":0,"maxPendingMessagesAcrossPartitions":0,"messageRoutingMode":"RoundRobinPartition","hashingScheme":"JavaStringHash","cryptoFailureAction":"FAIL","batchingMaxPublishDelayMicros":1000,"batchingPartitionSwitchFrequencyByPublishDelay":10,"batchingMaxMessages":1000,"batchingMaxBytes":131072,"batchingEnabled":true,"chunkingEnabled":false,"chunkMaxMessageSize":-1,"encryptionKeys":[],"compressionType":"NONE","initialSequenceId":null,"autoUpdatePartitions":true,"autoUpdatePartitionsIntervalSeconds":60,"multiSchema":true,"accessMode":"Shared","lazyStartPartitionedProducers":false,"properties":{},"initialSubscriptionName":null}
11:18:06.611 [ZScheduler-Worker-4] INFO o.a.p.c.i.ProducerStatsRecorderImpl
- Pulsar client config:
{"serviceUrl":"pulsar://localhost:6650","authPluginClassName":null,"authParams":null,"authParamMap":null,"operationTimeoutMs":3,"lookupTimeoutMs":3,"statsIntervalSeconds":60,"numIoThreads":12,"numListenerThreads":12,"connectionsPerBroker":1,"connectionMaxIdleSeconds":180,"useTcpNoDelay":true,"useTls":false,"tlsKeyFilePath":null,"tlsCertificateFilePath":null,"tlsTrustCertsFilePath":null,"tlsAllowInsecureConnection":false,"tlsHostnameVerificationEnable":false,"concurrentLookupRequest":5000,"maxLookupRequest":5,"maxLookupRedirects":20,"maxNumberOfRejectedRequestPerConnection":50,"keepAliveIntervalSeconds":30,"connectionTimeoutMs":1,"requestTimeoutMs":6,"readTimeoutMs":6,"autoCertRefreshSeconds":300,"initialBackoffIntervalNanos":1,"maxBackoffIntervalNanos":600,"enableBusyWait":false,"listenerName":null,"useKeyStoreTls":false,"sslProvider":null,"t
lsKeyStoreType":"JKS","tlsKeyStorePath":null,"tlsKeyStorePassword":null,"tlsTrustStoreType":"JKS","tlsTrustStorePath":null,"tlsTrustStorePassword":null,"tlsCiphers":[],"tlsProtocols":[],"memoryLimitBytes":67108864,"proxyServiceUrl":null,"proxyProtocol":null,"enableTransaction":false,"dnsLookupBindAddress":null,"dnsLookupBindPort":0,"dnsServerAddresses":[],"socks5ProxyAddress":null,"socks5ProxyUsername":null,"socks5ProxyPassword":null,"description":null}
11:18:06.622 [pulsar-client-io-3-3] INFO
o.a.pulsar.client.impl.ProducerImpl -
[persistent://new-tenant/new-namespace/topic-a] [null] Creating producer on cnx
[id: 0xffcc2c45, L:/127.0.0.1:50490 - R:localhost/127.0.0.1:6650]
11:18:06.683 [pulsar-client-io-3-3] INFO
o.a.pulsar.client.impl.ProducerImpl -
Re: [I] Topic creation not working properly [pulsar]
ragaur-tibco commented on issue #22527: URL: https://github.com/apache/pulsar/issues/22527#issuecomment-2065901746 > @ragaur-tibco > > > I tried below configuration > > if RegexSubscriptionMode = AllTopics > > not able to consume any messages > > If RegexSubscriptionMode = NonPartitionedOnly > > partition topics received > > and if RegexSubscriptionMode = PartitionOnly > > non partition topics received > > You probably mean `PersistentOnly` and `NonPersistentOnly` here. Sorry @visortelle With `PersistentOnly` working fine but with `NonPersistentOnly` it is not working as expected and with the `AllTopics` if I tried to get the non-persistent topic then I face some issue but with persistent I was able to consume messages -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Topic creation not working properly [pulsar]
visortelle commented on issue #22527: URL: https://github.com/apache/pulsar/issues/22527#issuecomment-2065895620 @ragaur-tibco > I tried below configuration if RegexSubscriptionMode = AllTopics not able to consume any messages If RegexSubscriptionMode = NonPartitionedOnly partition topics received and if RegexSubscriptionMode = PartitionOnly non partition topics received You probably mean `PersistentOnly` and `NonPersistentOnly` here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Topic creation not working properly [pulsar]
ragaur-tibco commented on issue #22527: URL: https://github.com/apache/pulsar/issues/22527#issuecomment-2065885718 @visortelle I tried below configuration if RegexSubscriptionMode = AllTopics not able to consume any messages If RegexSubscriptionMode = NonPartitionedOnly able to receive partition topics and if RegexSubscriptionMode = PartitionOnly able to receive non partition topics -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Topic creation not working properly [pulsar]
visortelle commented on issue #22527: URL: https://github.com/apache/pulsar/issues/22527#issuecomment-2065863301 @ragaur-tibco I see. --- > I don't see the Received ... From topic ... log messages on the first run when topics do not exist yet. ```scala .patternAutoDiscoveryPeriod(1) ``` Ah, ok. It is implemented with polling. I thought it uses some notification events sent to consumers to update the topics list. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Topic creation not working properly [pulsar]
ragaur-tibco commented on issue #22527: URL: https://github.com/apache/pulsar/issues/22527#issuecomment-2065831557 Hi @visortelle I have already created a different ticket for this link: https://github.com/apache/pulsar/issues/22529 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Topic creation not working properly [pulsar]
visortelle commented on issue #22527: URL: https://github.com/apache/pulsar/issues/22527#issuecomment-2065828143 New observations: - I don't see messages in logs on the first run when topics do not exist yet. - I don't see messages from non-persistent topics (`topic-c`, `topic-d`). Changing `RegexSubscriptionMode.AllTopics` => `RegexSubscriptionMode.NonPersistentOnly` doesn't help. ``` "persistent://new-tenant/new-namespace/topic-a", "persistent://new-tenant/new-namespace/topic-b", "non-persistent://new-tenant/new-namespace/topic-c", "non-persistent://new-tenant/new-namespace/topic-d", ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
