Re: [PR] [fix][broker] Introduce the last sent position to fix message ordering issues in Key_Shared (PIP-282) [pulsar]
equanz commented on PR #21953: URL: https://github.com/apache/pulsar/pull/21953#issuecomment-2159814582 Hi @codelipenghui . Do you have any comments? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][broker] The topic might reference a closed ledger [pulsar]
lhotari commented on code in PR #22860:
URL: https://github.com/apache/pulsar/pull/22860#discussion_r1634202683
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java:
##
@@ -1283,7 +1288,7 @@ private CompletableFuture>
createNonPersistentTopic(String topic
}).exceptionally(ex -> {
log.warn("Replication check failed. Removing topic from topics
list {}, {}", topic, ex.getCause());
nonPersistentTopic.stopReplProducers().whenComplete((v,
exception) -> {
-pulsar.getExecutor().execute(() -> topics.remove(topic,
topicFuture));
+pulsar.getExecutor().execute(() -> topics.remove(topic));
Review Comment:
> I have not found any reason for it.
There are a few issues that @poorbarcode has fixed by removing the specific
value instead of removing by key.
Perhaps he could share more details.
> However, in reality, if you want to compare topic futures, it is easy to
make mistakes in deletion.
That's true. The compiler won't catch the mistakes, but we should have tests
to cover that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [fix][io] Upgrade Debezium oracle connector version to 2.5.4.Final [pulsar]
lhotari commented on code in PR #22813:
URL: https://github.com/apache/pulsar/pull/22813#discussion_r1634194181
##
pulsar-io/debezium/oracle/pom.xml:
##
@@ -39,16 +39,49 @@
provided
+
+ io.debezium
+ debezium-connector-oracle
+ ${debezium.oracle.version}
+
+
+
+ io.debezium
+ debezium-core
+ ${debezium.oracle.version}
+
Review Comment:
I don't think it's a great idea to have a separate version of debezium-core
when Oracle connector is used. This will cause class version conflicts as we
have seen before. Please see #22668 as an example of this.
I think that should be upgrading Debezium in all connectors at the same time
to avoid class version conflicts and the confusion caused for maintenance when
multiple versions of Debezium are used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [fix][io] Upgrade Debezium oracle connector version to 2.5.4.Final [pulsar]
lhotari commented on code in PR #22813:
URL: https://github.com/apache/pulsar/pull/22813#discussion_r1634194181
##
pulsar-io/debezium/oracle/pom.xml:
##
@@ -39,16 +39,49 @@
provided
+
+ io.debezium
+ debezium-connector-oracle
+ ${debezium.oracle.version}
+
+
+
+ io.debezium
+ debezium-core
+ ${debezium.oracle.version}
+
Review Comment:
I don't think it's a great idea to have a separate version of debezium-core
when Oracle connector is used. This will cause class version conflicts as we
have seen before. Please see #22668 as an example of this.
I think that should be upgrading Debezium in all connectors at the same time
to avoid class version conflicts and the confusion caused for maintenance when
multiple versions of Debezium is used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [fix] [broker] response not-found error if topic does not exist when calling getPartitionedTopicMetadata [pulsar]
poorbarcode commented on code in PR #22838:
URL: https://github.com/apache/pulsar/pull/22838#discussion_r1634193462
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/TopicExistsInfo.java:
##
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.namespace;
+
+import io.netty.util.Recycler;
+import lombok.Getter;
+import org.apache.pulsar.common.policies.data.TopicType;
+
+public class TopicExistsInfo {
+
+private static final Recycler RECYCLER = new Recycler<>()
{
+@Override
+protected TopicExistsInfo newObject(Handle handle) {
+return new TopicExistsInfo(handle);
+}
+};
+
+private static TopicExistsInfo nonPartitionedExists = new
TopicExistsInfo(null);
+static {
+nonPartitionedExists.exists = true;
+nonPartitionedExists.topicType = TopicType.NON_PARTITIONED;
+nonPartitionedExists.partitions = null;
+}
+
+private static TopicExistsInfo notExists = new TopicExistsInfo(null);
+static {
+notExists.exists = false;
+notExists.topicType = TopicType.NON_PARTITIONED;
+notExists.partitions = null;
+}
Review Comment:
Improved
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/TopicExistsInfo.java:
##
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.namespace;
+
+import io.netty.util.Recycler;
+import lombok.Getter;
+import org.apache.pulsar.common.policies.data.TopicType;
+
+public class TopicExistsInfo {
+
+private static final Recycler RECYCLER = new Recycler<>()
{
+@Override
+protected TopicExistsInfo newObject(Handle handle) {
+return new TopicExistsInfo(handle);
+}
+};
+
+private static TopicExistsInfo nonPartitionedExists = new
TopicExistsInfo(null);
+static {
+nonPartitionedExists.exists = true;
+nonPartitionedExists.topicType = TopicType.NON_PARTITIONED;
+nonPartitionedExists.partitions = null;
+}
Review Comment:
Improved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [fix] [broker] response not-found error if topic does not exist when calling getPartitionedTopicMetadata [pulsar]
poorbarcode commented on code in PR #22838:
URL: https://github.com/apache/pulsar/pull/22838#discussion_r1634193329
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/TopicExistsInfo.java:
##
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.namespace;
+
+import io.netty.util.Recycler;
+import lombok.Getter;
+import org.apache.pulsar.common.policies.data.TopicType;
+
+public class TopicExistsInfo {
+
+private static final Recycler RECYCLER = new Recycler<>()
{
+@Override
+protected TopicExistsInfo newObject(Handle handle) {
+return new TopicExistsInfo(handle);
+}
+};
+
+private static TopicExistsInfo nonPartitionedExists = new
TopicExistsInfo(null);
+static {
+nonPartitionedExists.exists = true;
+nonPartitionedExists.topicType = TopicType.NON_PARTITIONED;
+nonPartitionedExists.partitions = null;
+}
+
+private static TopicExistsInfo notExists = new TopicExistsInfo(null);
+static {
+notExists.exists = false;
+notExists.topicType = TopicType.NON_PARTITIONED;
+notExists.partitions = null;
+}
+
+public static TopicExistsInfo partitionedExists(Integer partitions){
+return newInstance(true, TopicType.PARTITIONED, partitions);
+}
+
+public static TopicExistsInfo nonPartitionedExists(){
+return nonPartitionedExists;
+}
+
+public static TopicExistsInfo notExists(){
Review Comment:
Renamed
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/TopicExistsInfo.java:
##
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.namespace;
+
+import io.netty.util.Recycler;
+import lombok.Getter;
+import org.apache.pulsar.common.policies.data.TopicType;
+
+public class TopicExistsInfo {
+
+private static final Recycler RECYCLER = new Recycler<>()
{
+@Override
+protected TopicExistsInfo newObject(Handle handle) {
+return new TopicExistsInfo(handle);
+}
+};
+
+private static TopicExistsInfo nonPartitionedExists = new
TopicExistsInfo(null);
+static {
+nonPartitionedExists.exists = true;
+nonPartitionedExists.topicType = TopicType.NON_PARTITIONED;
+nonPartitionedExists.partitions = null;
+}
+
+private static TopicExistsInfo notExists = new TopicExistsInfo(null);
+static {
+notExists.exists = false;
+notExists.topicType = TopicType.NON_PARTITIONED;
+notExists.partitions = null;
+}
+
+public static TopicExistsInfo partitionedExists(Integer partitions){
+return newInstance(true, TopicType.PARTITIONED, partitions);
+}
+
+public static TopicExistsInfo nonPartitionedExists(){
Review Comment:
Renamed
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/TopicExistsInfo.java:
##
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *
Re: [PR] [fix] [broker] response not-found error if topic does not exist when calling getPartitionedTopicMetadata [pulsar]
poorbarcode commented on code in PR #22838:
URL: https://github.com/apache/pulsar/pull/22838#discussion_r1634193111
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/TopicExistsInfo.java:
##
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.namespace;
+
+import io.netty.util.Recycler;
+import lombok.Getter;
+import org.apache.pulsar.common.policies.data.TopicType;
+
+public class TopicExistsInfo {
+
+private static final Recycler RECYCLER = new Recycler<>()
{
+@Override
+protected TopicExistsInfo newObject(Handle handle) {
+return new TopicExistsInfo(handle);
+}
+};
+
+private static TopicExistsInfo nonPartitionedExists = new
TopicExistsInfo(null);
+static {
+nonPartitionedExists.exists = true;
+nonPartitionedExists.topicType = TopicType.NON_PARTITIONED;
+nonPartitionedExists.partitions = null;
+}
+
+private static TopicExistsInfo notExists = new TopicExistsInfo(null);
+static {
+notExists.exists = false;
+notExists.topicType = TopicType.NON_PARTITIONED;
+notExists.partitions = null;
+}
+
+public static TopicExistsInfo partitionedExists(Integer partitions){
+return newInstance(true, TopicType.PARTITIONED, partitions);
+}
+
+public static TopicExistsInfo nonPartitionedExists(){
+return nonPartitionedExists;
+}
+
+public static TopicExistsInfo notExists(){
+return notExists;
+}
+
+private static TopicExistsInfo newInstance(boolean exists, TopicType
topicType, Integer partitions){
+TopicExistsInfo info = RECYCLER.get();
+info.exists = exists;
+info.topicType = topicType;
+if (topicType == null) {
+throw new IllegalArgumentException("The param topicType can not be
null when creating a TopicExistsInfo"
++ " obj.");
+}
+if (topicType.equals(TopicType.PARTITIONED)) {
+if (partitions == null || partitions.intValue() < 1) {
+throw new IllegalArgumentException("The param partitions can
not be null or less than 1 when creating"
++ " a partitioned TopicExistsInfo obj.");
+}
+info.partitions = partitions.intValue();
+} else {
+if (partitions != null) {
+throw new IllegalArgumentException("The param partitions must
be null when creating a non-partitioned"
++ " TopicExistsInfo obj.");
+}
+}
+return info;
+}
+
+private final Recycler.Handle handle;
+
+@Getter
+private TopicType topicType;
Review Comment:
Removed the field `topicType`
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/TopicExistsInfo.java:
##
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.namespace;
+
+import io.netty.util.Recycler;
+import lombok.Getter;
+import org.apache.pulsar.common.policies.data.TopicType;
+
+public class TopicExistsInfo {
+
+private static final Recycler RECYCLER = new Recycler<>()
{
+@Override
+protected TopicExistsInfo newObject(Handle handle) {
+return new TopicExistsInfo(handle);
+}
+};
+
+private static TopicExistsInfo nonPartitionedExists = new
Re: [PR] [fix] [broker] response not-found error if topic does not exist when calling getPartitionedTopicMetadata [pulsar]
poorbarcode commented on code in PR #22838:
URL: https://github.com/apache/pulsar/pull/22838#discussion_r1634192813
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/TopicExistsInfo.java:
##
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.namespace;
+
+import io.netty.util.Recycler;
+import lombok.Getter;
+import org.apache.pulsar.common.policies.data.TopicType;
+
+public class TopicExistsInfo {
+
+private static final Recycler RECYCLER = new Recycler<>()
{
+@Override
+protected TopicExistsInfo newObject(Handle handle) {
+return new TopicExistsInfo(handle);
+}
+};
+
+private static TopicExistsInfo nonPartitionedExists = new
TopicExistsInfo(null);
+static {
+nonPartitionedExists.exists = true;
+nonPartitionedExists.topicType = TopicType.NON_PARTITIONED;
+nonPartitionedExists.partitions = null;
+}
+
+private static TopicExistsInfo notExists = new TopicExistsInfo(null);
+static {
+notExists.exists = false;
+notExists.topicType = TopicType.NON_PARTITIONED;
+notExists.partitions = null;
+}
+
+public static TopicExistsInfo partitionedExists(Integer partitions){
+return newInstance(true, TopicType.PARTITIONED, partitions);
+}
+
+public static TopicExistsInfo nonPartitionedExists(){
+return nonPartitionedExists;
+}
+
+public static TopicExistsInfo notExists(){
+return notExists;
+}
+
+private static TopicExistsInfo newInstance(boolean exists, TopicType
topicType, Integer partitions){
+TopicExistsInfo info = RECYCLER.get();
+info.exists = exists;
+info.topicType = topicType;
+if (topicType == null) {
+throw new IllegalArgumentException("The param topicType can not be
null when creating a TopicExistsInfo"
++ " obj.");
+}
+if (topicType.equals(TopicType.PARTITIONED)) {
+if (partitions == null || partitions.intValue() < 1) {
+throw new IllegalArgumentException("The param partitions can
not be null or less than 1 when creating"
++ " a partitioned TopicExistsInfo obj.");
+}
+info.partitions = partitions.intValue();
+} else {
+if (partitions != null) {
+throw new IllegalArgumentException("The param partitions must
be null when creating a non-partitioned"
++ " TopicExistsInfo obj.");
+}
+}
+return info;
+}
+
+private final Recycler.Handle handle;
+
+@Getter
+private TopicType topicType;
+@Getter
+private Integer partitions;
+@Getter
+private boolean exists;
+
+private TopicExistsInfo(Recycler.Handle handle) {
+this.handle = handle;
+}
+
+public void recycle(){
Review Comment:
Fixed
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/TopicExistsInfo.java:
##
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.namespace;
+
+import io.netty.util.Recycler;
+import lombok.Getter;
+import org.apache.pulsar.common.policies.data.TopicType;
+
+public class TopicExistsInfo {
+
+private static final Recycler RECYCLER = new Recycler<>()
{
+@Override
+
Re: [PR] [improve][cli] Use LoadManagerReport instead of Object [pulsar]
lhotari commented on PR #22850: URL: https://github.com/apache/pulsar/pull/22850#issuecomment-2159796740 Since this code is touched, it might be useful to make use of `loadReportType` in detecting the type of the object that is stored in JSON: https://github.com/apache/pulsar/blob/c326d8e2203b6e9be37f4f2066fd7e90a9b9fb54/pulsar-testclient/src/main/java/org/apache/pulsar/testclient/BrokerMonitor.java#L315-L328 It could be useful to refactor BrokerMonitor to use Jackson instead of Gson. Mixing Jackson and Gson in the code base is something we want to get rid of eventually. `loadReportType` is handled in Pulsar's Jackson ObjectMapper config: https://github.com/apache/pulsar/blob/c326d8e2203b6e9be37f4f2066fd7e90a9b9fb54/pulsar-common/src/main/java/org/apache/pulsar/common/util/ObjectMapperFactory.java#L263-L271 When LoadManagerReport is requested, it will choose the correct implementation class based on the loadReportType field (however, it looks that shaded classnames aren't properly handled). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch master updated: [fix][misc] Topic name from persistence name should decode local name (#22879)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new c326d8e2203 [fix][misc] Topic name from persistence name should decode
local name (#22879)
c326d8e2203 is described below
commit c326d8e2203b6e9be37f4f2066fd7e90a9b9fb54
Author: 萧易客
AuthorDate: Tue Jun 11 12:46:04 2024 +0800
[fix][misc] Topic name from persistence name should decode local name
(#22879)
---
.../src/main/java/org/apache/pulsar/common/naming/TopicName.java| 5 ++---
.../test/java/org/apache/pulsar/common/naming/TopicNameTest.java| 6 ++
2 files changed, 8 insertions(+), 3 deletions(-)
diff --git
a/pulsar-common/src/main/java/org/apache/pulsar/common/naming/TopicName.java
b/pulsar-common/src/main/java/org/apache/pulsar/common/naming/TopicName.java
index eebca0e0d72..e051e01495d 100644
--- a/pulsar-common/src/main/java/org/apache/pulsar/common/naming/TopicName.java
+++ b/pulsar-common/src/main/java/org/apache/pulsar/common/naming/TopicName.java
@@ -358,17 +358,16 @@ public class TopicName implements ServiceUnitId {
String localName;
if (parts.size() == 4) {
tenant = parts.get(0);
-cluster = null;
namespacePortion = parts.get(1);
domain = parts.get(2);
-localName = parts.get(3);
+localName = Codec.decode(parts.get(3));
return String.format("%s://%s/%s/%s", domain, tenant,
namespacePortion, localName);
} else if (parts.size() == 5) {
tenant = parts.get(0);
cluster = parts.get(1);
namespacePortion = parts.get(2);
domain = parts.get(3);
-localName = parts.get(4);
+localName = Codec.decode(parts.get(4));
return String.format("%s://%s/%s/%s/%s", domain, tenant, cluster,
namespacePortion, localName);
} else {
throw new IllegalArgumentException("Invalid managedLedger name: "
+ mlName);
diff --git
a/pulsar-common/src/test/java/org/apache/pulsar/common/naming/TopicNameTest.java
b/pulsar-common/src/test/java/org/apache/pulsar/common/naming/TopicNameTest.java
index 835045f9167..485bea3f1ad 100644
---
a/pulsar-common/src/test/java/org/apache/pulsar/common/naming/TopicNameTest.java
+++
b/pulsar-common/src/test/java/org/apache/pulsar/common/naming/TopicNameTest.java
@@ -267,6 +267,12 @@ public class TopicNameTest {
} catch (IllegalArgumentException e) {
// Exception is expected.
}
+
+// case5: local name with special characters e.g. a:b:c
+String topicName = "persistent://tenant/namespace/a:b:c";
+String persistentNamingEncoding =
"tenant/namespace/persistent/a%3Ab%3Ac";
+assertEquals(TopicName.get(topicName).getPersistenceNamingEncoding(),
persistentNamingEncoding);
+
assertEquals(TopicName.fromPersistenceNamingEncoding(persistentNamingEncoding),
topicName);
}
Re: [PR] [fix][misc] Topic name from persistence name should decode local name [pulsar]
lhotari merged PR #22879: URL: https://github.com/apache/pulsar/pull/22879 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix] [broker] response not-found error if topic does not exist when calling getPartitionedTopicMetadata [pulsar]
codelipenghui commented on code in PR #22838:
URL: https://github.com/apache/pulsar/pull/22838#discussion_r1634141124
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/lookup/TopicLookupBase.java:
##
@@ -70,13 +70,14 @@ protected CompletableFuture
internalLookupTopicAsync(final TopicName
// Currently, it's hard to check the
non-persistent-non-partitioned topic, because it only exists
// in the broker, it doesn't have metadata. If the topic
is non-persistent and non-partitioned,
// we'll return the true flag.
-CompletableFuture existFuture =
(!topicName.isPersistent() && !topicName.isPartitioned())
-? CompletableFuture.completedFuture(true)
-:
pulsar().getNamespaceService().checkTopicExists(topicName)
-.thenCompose(exists -> exists ?
CompletableFuture.completedFuture(true)
-:
pulsar().getBrokerService().isAllowAutoTopicCreationAsync(topicName));
-
-return existFuture;
+return
pulsar().getNamespaceService().checkTopicExists(topicName,
true).thenCompose(info -> {
+boolean exists = info.isExists();
+info.recycle();
+if (exists) {
+return CompletableFuture.completedFuture(true);
+}
+return
pulsar().getBrokerService().isAllowAutoTopicCreationAsync(topicName);
+});
Review Comment:
I would like to suggest to remove the param of
`assumedNonPartitionedNonPersistentTopicAlwaysExists` in
NamespaceService#checkTopicExists(). Here is the only place that set this param
to true.
Instead, we can just return true here if the topic is a non-persistent topic.
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/TopicExistsInfo.java:
##
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.namespace;
+
+import io.netty.util.Recycler;
+import lombok.Getter;
+import org.apache.pulsar.common.policies.data.TopicType;
+
+public class TopicExistsInfo {
+
+private static final Recycler RECYCLER = new Recycler<>()
{
+@Override
+protected TopicExistsInfo newObject(Handle handle) {
+return new TopicExistsInfo(handle);
+}
+};
+
+private static TopicExistsInfo nonPartitionedExists = new
TopicExistsInfo(null);
+static {
+nonPartitionedExists.exists = true;
+nonPartitionedExists.topicType = TopicType.NON_PARTITIONED;
+nonPartitionedExists.partitions = null;
+}
+
+private static TopicExistsInfo notExists = new TopicExistsInfo(null);
+static {
+notExists.exists = false;
+notExists.topicType = TopicType.NON_PARTITIONED;
+notExists.partitions = null;
+}
+
+public static TopicExistsInfo partitionedExists(Integer partitions){
Review Comment:
```suggestion
public static TopicExistsInfo newPartitionedTopicExists(Integer
partitions){
```
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/TopicExistsInfo.java:
##
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package
Re: [PR] [fix] [broker] fix create LoadSheddingStrategy instance. [pulsar]
codecov-commenter commented on PR #22827: URL: https://github.com/apache/pulsar/pull/22827#issuecomment-2159749865 ## [Codecov](https://app.codecov.io/gh/apache/pulsar/pull/22827?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report Attention: Patch coverage is `0%` with `3 lines` in your changes missing coverage. Please review. > Project coverage is 73.24%. Comparing base [(`bbc6224`)](https://app.codecov.io/gh/apache/pulsar/commit/bbc62245c5ddba1de4b1e7cee4ab49334bc36277?dropdown=coverage=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) to head [(`5cf83d1`)](https://app.codecov.io/gh/apache/pulsar/commit/5cf83d104a5e6fe2bfa8ba3d612f698901d615a6?dropdown=coverage=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache). > Report is 369 commits behind head on master. Additional details and impacted files [](https://app.codecov.io/gh/apache/pulsar/pull/22827?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) ```diff @@ Coverage Diff @@ ## master #22827 +/- ## - Coverage 73.57% 73.24% -0.34% - Complexity3262432638 +14 Files 1877 1889 +12 Lines139502 141744+2242 Branches 1529915554 +255 + Hits 102638 103816+1178 - Misses2890829908+1000 - Partials 7956 8020 +64 ``` | [Flag](https://app.codecov.io/gh/apache/pulsar/pull/22827/flags?src=pr=flags_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Coverage Δ | | |---|---|---| | [inttests](https://app.codecov.io/gh/apache/pulsar/pull/22827/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `27.44% <0.00%> (+2.85%)` | :arrow_up: | | [systests](https://app.codecov.io/gh/apache/pulsar/pull/22827/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `24.67% <0.00%> (+0.35%)` | :arrow_up: | | [unittests](https://app.codecov.io/gh/apache/pulsar/pull/22827/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `72.26% <0.00%> (-0.58%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files](https://app.codecov.io/gh/apache/pulsar/pull/22827?dropdown=coverage=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Coverage Δ | | |---|---|---| | [...roker/loadbalance/impl/ModularLoadManagerImpl.java](https://app.codecov.io/gh/apache/pulsar/pull/22827?src=pr=tree=pulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Floadbalance%2Fimpl%2FModularLoadManagerImpl.java_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9sb2FkYmFsYW5jZS9pbXBsL01vZHVsYXJMb2FkTWFuYWdlckltcGwuamF2YQ==) | `84.20% <0.00%> (-0.79%)` | :arrow_down: | ... and [373 files with indirect coverage changes](https://app.codecov.io/gh/apache/pulsar/pull/22827/indirect-changes?src=pr=tree-more_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix] [broker] response not-found error if topic does not exist when calling getPartitionedTopicMetadata [pulsar]
codecov-commenter commented on PR #22838: URL: https://github.com/apache/pulsar/pull/22838#issuecomment-2159743315 ## [Codecov](https://app.codecov.io/gh/apache/pulsar/pull/22838?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report Attention: Patch coverage is `80.0%` with `35 lines` in your changes missing coverage. Please review. > Project coverage is 73.26%. Comparing base [(`bbc6224`)](https://app.codecov.io/gh/apache/pulsar/commit/bbc62245c5ddba1de4b1e7cee4ab49334bc36277?dropdown=coverage=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) to head [(`3e061cc`)](https://app.codecov.io/gh/apache/pulsar/commit/3e061cc53c79fe62f1aa5681aec82e502b10b06b?dropdown=coverage=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache). > Report is 369 commits behind head on master. Additional details and impacted files [](https://app.codecov.io/gh/apache/pulsar/pull/22838?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) ```diff @@ Coverage Diff @@ ## master #22838 +/- ## - Coverage 73.57% 73.26% -0.31% - Complexity3262433033 +409 Files 1877 1892 +15 Lines139502 142015+2513 Branches 1529915576 +277 + Hits 102638 104047+1409 - Misses2890829949+1041 - Partials 7956 8019 +63 ``` | [Flag](https://app.codecov.io/gh/apache/pulsar/pull/22838/flags?src=pr=flags_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Coverage Δ | | |---|---|---| | [inttests](https://app.codecov.io/gh/apache/pulsar/pull/22838/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `27.35% <54.28%> (+2.76%)` | :arrow_up: | | [systests](https://app.codecov.io/gh/apache/pulsar/pull/22838/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `24.69% <24.57%> (+0.37%)` | :arrow_up: | | [unittests](https://app.codecov.io/gh/apache/pulsar/pull/22838/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `72.30% <80.00%> (-0.55%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files](https://app.codecov.io/gh/apache/pulsar/pull/22838?dropdown=coverage=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Coverage Δ | | |---|---|---| | [...pulsar/broker/admin/impl/PersistentTopicsBase.java](https://app.codecov.io/gh/apache/pulsar/pull/22838?src=pr=tree=pulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fadmin%2Fimpl%2FPersistentTopicsBase.java_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi9pbXBsL1BlcnNpc3RlbnRUb3BpY3NCYXNlLmphdmE=) | `69.73% <100.00%> (+4.27%)` | :arrow_up: | | [...he/pulsar/broker/admin/v2/NonPersistentTopics.java](https://app.codecov.io/gh/apache/pulsar/pull/22838?src=pr=tree=pulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fadmin%2Fv2%2FNonPersistentTopics.java_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi92Mi9Ob25QZXJzaXN0ZW50VG9waWNzLmphdmE=) | `63.87% <100.00%> (+2.40%)` | :arrow_up: | | [...g/apache/pulsar/broker/lookup/TopicLookupBase.java](https://app.codecov.io/gh/apache/pulsar/pull/22838?src=pr=tree=pulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Flookup%2FTopicLookupBase.java_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9sb29rdXAvVG9waWNMb29rdXBCYXNlLmphdmE=) | `75.84% <100.00%> (+8.42%)` | :arrow_up: | |
Re: [PR] [improve] [pip] PIP-358: let resource weight work for OverloadShedder, LeastLongTermMessageRate, ModularLoadManagerImpl. [pulsar]
thetumbled commented on PR #22889: URL: https://github.com/apache/pulsar/pull/22889#issuecomment-2159729344 PTAL, thanks. @Demogorgon314 @heesung-sn @codelipenghui @BewareMyPower @Technoboy- -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix] [broker] fix create LoadSheddingStrategy instance. [pulsar]
thetumbled commented on PR #22827: URL: https://github.com/apache/pulsar/pull/22827#issuecomment-2159725856 /pulsarbot rerun-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [improve] [broker] let resource weight work for OverloadShedder, LeastLongTermMessageRate, ModularLoadManagerImpl. [pulsar]
thetumbled opened a new pull request, #22889: URL: https://github.com/apache/pulsar/pull/22889 ### Motivation Initially, we introduce `loadBalancerCPUResourceWeight`, `loadBalancerBandwidthInResourceWeight`, `loadBalancerBandwidthOutResourceWeight`, `loadBalancerMemoryResourceWeight`, `loadBalancerDirectMemoryResourceWeight` in `ThresholdShedder` to control the resource weight for different resources when calculating the load of the broker. Then we let it work for `LeastResourceUsageWithWeight` for better bundle placement policy. But https://github.com/apache/pulsar/pull/19559 and https://github.com/apache/pulsar/pull/21168 have point out that the actual load of the broker is not related to the memory usage and direct memory usage, thus we have changed the default value of `loadBalancerMemoryResourceWeight`, `loadBalancerDirectMemoryResourceWeight` to 0.0. There are still some places where memory usage and direct memory usage are used to calculate the load of the broker, such as `OverloadShedder`, `LeastLongTermMessageRate`, `ModularLoadManagerImpl`. We should let the resource weight work for these places so that we can set the resource weight to 0.0 to avoid the impact of memory usage and direct memory usage on the load of the broker. ### Modifications - Let resource weight work for `OverloadShedder`, `LeastLongTermMessageRate`, `ModularLoadManagerImpl`. ### Verifying this change - [ ] Make sure that the change passes the CI checks. *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. ### Does this pull request potentially affect one of the following parts: *If the box was checked, please highlight the changes* - [ ] Dependencies (add or upgrade a dependency) - [ ] The public API - [ ] The schema - [ ] The default values of configurations - [ ] The threading model - [ ] The binary protocol - [ ] The REST endpoints - [ ] The admin CLI options - [ ] The metrics - [ ] Anything that affects deployment ### Documentation - [ ] `doc` - [ ] `doc-required` - [ ] `doc-not-needed` - [ ] `doc-complete` ### Matching PR in forked repository PR in forked repository: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Add sdkman support [pulsar]
nodece closed issue #22298: Add sdkman support URL: https://github.com/apache/pulsar/issues/22298 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][broker] The topic might reference a closed ledger [pulsar]
codelipenghui commented on code in PR #22860:
URL: https://github.com/apache/pulsar/pull/22860#discussion_r1634118133
##
pulsar-broker/src/test/java/org/apache/pulsar/client/api/OrphanPersistentTopicTest.java:
##
@@ -108,6 +114,67 @@ public void testNoOrphanTopicAfterCreateTimeout() throws
Exception {
pulsar.getConfig().setTopicLoadTimeoutSeconds(originalTopicLoadTimeoutSeconds);
}
+@Test
+public void testCloseLedgerThatTopicAfterCreateTimeout() throws Exception {
Review Comment:
@shibd Yes, I just copied the tests to the master branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [improve] [broker] make resource weight work for OverloadShedder, LeastLongTermMessageRate, ModularLoadManagerImpl. [pulsar]
thetumbled commented on PR #22888: URL: https://github.com/apache/pulsar/pull/22888#issuecomment-2159695049 /pulsarbot rerun-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [improve] [broker] make resource weight work for OverloadShedder, LeastLongTermMessageRate, ModularLoadManagerImpl. [pulsar]
thetumbled opened a new pull request, #22888: URL: https://github.com/apache/pulsar/pull/22888 PIP: #xyz ### Motivation ### Modifications ### Verifying this change - [ ] Make sure that the change passes the CI checks. *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Does this pull request potentially affect one of the following parts: *If the box was checked, please highlight the changes* - [ ] Dependencies (add or upgrade a dependency) - [ ] The public API - [ ] The schema - [ ] The default values of configurations - [ ] The threading model - [ ] The binary protocol - [ ] The REST endpoints - [ ] The admin CLI options - [ ] The metrics - [ ] Anything that affects deployment ### Documentation - [ ] `doc` - [ ] `doc-required` - [ ] `doc-not-needed` - [ ] `doc-complete` ### Matching PR in forked repository PR in forked repository: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][broker] The topic might reference a closed ledger [pulsar]
shibd commented on code in PR #22860:
URL: https://github.com/apache/pulsar/pull/22860#discussion_r1634112064
##
pulsar-broker/src/test/java/org/apache/pulsar/client/api/OrphanPersistentTopicTest.java:
##
@@ -108,6 +114,67 @@ public void testNoOrphanTopicAfterCreateTimeout() throws
Exception {
pulsar.getConfig().setTopicLoadTimeoutSeconds(originalTopicLoadTimeoutSeconds);
}
+@Test
+public void testCloseLedgerThatTopicAfterCreateTimeout() throws Exception {
Review Comment:
Interesting. Are you copying the tests to the master branch to test it?
If use this PR branch and revert that fix code, it always fails.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [fix][broker] The topic might reference a closed ledger [pulsar]
codelipenghui commented on code in PR #22860:
URL: https://github.com/apache/pulsar/pull/22860#discussion_r1634094976
##
pulsar-broker/src/test/java/org/apache/pulsar/client/api/OrphanPersistentTopicTest.java:
##
@@ -108,6 +114,67 @@ public void testNoOrphanTopicAfterCreateTimeout() throws
Exception {
pulsar.getConfig().setTopicLoadTimeoutSeconds(originalTopicLoadTimeoutSeconds);
}
+@Test
+public void testCloseLedgerThatTopicAfterCreateTimeout() throws Exception {
Review Comment:
@shibd I tried this test without this fix. The test can still get passed. Do
you have a test to reproduce the issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
(pulsar) branch branch-2.11 updated: [improve] [client] improve the class GetTopicsResult (#22766)
This is an automated email from the ASF dual-hosted git repository.
yubiao pushed a commit to branch branch-2.11
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-2.11 by this push:
new 9f1118cb892 [improve] [client] improve the class GetTopicsResult
(#22766)
9f1118cb892 is described below
commit 9f1118cb892ec4bd6e4e13a42a8d6adec95aa449
Author: fengyubiao
AuthorDate: Tue Jun 11 10:40:26 2024 +0800
[improve] [client] improve the class GetTopicsResult (#22766)
(cherry picked from commit 87a33399873ff1e9723a6ca3812cbf914d8c8eef)
---
.../pulsar/client/impl/LookupServiceTest.java | 128 +
.../client/impl/BinaryProtoLookupService.java | 14 +--
.../pulsar/client/impl/HttpLookupService.java | 13 +--
.../pulsar/common/lookup/GetTopicsResult.java | 106 +++--
4 files changed, 225 insertions(+), 36 deletions(-)
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/LookupServiceTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/LookupServiceTest.java
new file mode 100644
index 000..c41f0afd16f
--- /dev/null
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/LookupServiceTest.java
@@ -0,0 +1,128 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl;
+
+import static
org.apache.pulsar.common.api.proto.CommandGetTopicsOfNamespace.Mode;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertTrue;
+import java.util.Collection;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.BrokerTestUtil;
+import org.apache.pulsar.client.api.ProducerConsumerBase;
+import org.apache.pulsar.client.api.PulsarClient;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicName;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.DataProvider;
+import org.testng.annotations.Test;
+
+@Test(groups = "broker-admin")
+@Slf4j
+public class LookupServiceTest extends ProducerConsumerBase {
+
+private PulsarClientImpl clientWithHttpLookup;
+private PulsarClientImpl clientWitBinaryLookup;
+
+private boolean enableBrokerSideSubscriptionPatternEvaluation = true;
+private int subscriptionPatternMaxLength = 10_000;
+
+@Override
+@BeforeClass
+protected void setup() throws Exception {
+super.internalSetup();
+super.producerBaseSetup();
+clientWithHttpLookup =
+(PulsarClientImpl)
PulsarClient.builder().serviceUrl(pulsar.getWebServiceAddress()).build();
+clientWitBinaryLookup =
+(PulsarClientImpl)
PulsarClient.builder().serviceUrl(pulsar.getBrokerServiceUrl()).build();
+}
+
+@Override
+@AfterClass(alwaysRun = true)
+protected void cleanup() throws Exception {
+super.internalCleanup();
+if (clientWithHttpLookup != null) {
+clientWithHttpLookup.close();
+}
+if (clientWitBinaryLookup != null) {
+clientWitBinaryLookup.close();
+}
+}
+
+@Override
+protected void doInitConf() throws Exception {
+super.doInitConf();
+
conf.setEnableBrokerSideSubscriptionPatternEvaluation(enableBrokerSideSubscriptionPatternEvaluation);
+conf.setSubscriptionPatternMaxLength(subscriptionPatternMaxLength);
+}
+
+private LookupService getLookupService(boolean isUsingHttpLookup) {
+if (isUsingHttpLookup) {
+return clientWithHttpLookup.getLookup();
+} else {
+return clientWitBinaryLookup.getLookup();
+}
+}
+
+@DataProvider(name = "isUsingHttpLookup")
+public Object[][] isUsingHttpLookup() {
+return new Object[][]{
+{true},
+{false}
+};
+}
+
+@Test(dataProvider = "isUsingHttpLookup")
+public void testGetTopicsOfGetTopicsResult(boolean isUsingHttpLookup)
throws Exception {
+LookupService lookupService = getLookupService(isUsingHttpLookup);
+String nonPartitionedTopic =
(pulsar) branch branch-3.3 updated: [improve] [client] improve the class GetTopicsResult (#22766)
This is an automated email from the ASF dual-hosted git repository.
yubiao pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new 12df7ff4644 [improve] [client] improve the class GetTopicsResult
(#22766)
12df7ff4644 is described below
commit 12df7ff464416dd250eaaad579fe7139a8250223
Author: fengyubiao
AuthorDate: Thu May 30 16:42:26 2024 +0800
[improve] [client] improve the class GetTopicsResult (#22766)
(cherry picked from commit 87a33399873ff1e9723a6ca3812cbf914d8c8eef)
---
.../pulsar/client/impl/LookupServiceTest.java | 128 +
.../client/impl/BinaryProtoLookupService.java | 14 +--
.../pulsar/client/impl/HttpLookupService.java | 13 +--
.../pulsar/common/lookup/GetTopicsResult.java | 106 +++--
4 files changed, 225 insertions(+), 36 deletions(-)
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/LookupServiceTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/LookupServiceTest.java
new file mode 100644
index 000..59cb7ae03d0
--- /dev/null
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/LookupServiceTest.java
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl;
+
+import static
org.apache.pulsar.common.api.proto.CommandGetTopicsOfNamespace.Mode;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertTrue;
+import java.util.Collection;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.BrokerTestUtil;
+import org.apache.pulsar.client.api.ProducerConsumerBase;
+import org.apache.pulsar.client.api.PulsarClient;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicName;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.DataProvider;
+import org.testng.annotations.Test;
+
+@Test(groups = "broker-admin")
+@Slf4j
+public class LookupServiceTest extends ProducerConsumerBase {
+
+private PulsarClientImpl clientWithHttpLookup;
+private PulsarClientImpl clientWitBinaryLookup;
+
+private boolean enableBrokerSideSubscriptionPatternEvaluation = true;
+private int subscriptionPatternMaxLength = 10_000;
+
+@Override
+@BeforeClass
+protected void setup() throws Exception {
+super.internalSetup();
+super.producerBaseSetup();
+clientWithHttpLookup =
+(PulsarClientImpl)
PulsarClient.builder().serviceUrl(pulsar.getWebServiceAddress()).build();
+clientWitBinaryLookup =
+(PulsarClientImpl)
PulsarClient.builder().serviceUrl(pulsar.getBrokerServiceUrl()).build();
+}
+
+@Override
+@AfterClass(alwaysRun = true)
+protected void cleanup() throws Exception {
+super.internalCleanup();
+if (clientWithHttpLookup != null) {
+clientWithHttpLookup.close();
+}
+if (clientWitBinaryLookup != null) {
+clientWitBinaryLookup.close();
+}
+}
+
+@Override
+protected void doInitConf() throws Exception {
+super.doInitConf();
+
conf.setEnableBrokerSideSubscriptionPatternEvaluation(enableBrokerSideSubscriptionPatternEvaluation);
+conf.setSubscriptionPatternMaxLength(subscriptionPatternMaxLength);
+}
+
+private LookupService getLookupService(boolean isUsingHttpLookup) {
+if (isUsingHttpLookup) {
+return clientWithHttpLookup.getLookup();
+} else {
+return clientWitBinaryLookup.getLookup();
+}
+}
+
+@DataProvider(name = "isUsingHttpLookup")
+public Object[][] isUsingHttpLookup() {
+return new Object[][]{
+{true},
+{false}
+};
+}
+
+@Test(dataProvider = "isUsingHttpLookup")
+public void testGetTopicsOfGetTopicsResult(boolean isUsingHttpLookup)
throws Exception {
+LookupService lookupService = getLookupService(isUsingHttpLookup);
+String nonPartitionedTopic =
Re: [PR] [feat][broker] PIP-321 Introduce allowed-cluster at the namespace level [pulsar]
Demogorgon314 commented on code in PR #22378:
URL: https://github.com/apache/pulsar/pull/22378#discussion_r1634083595
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java:
##
@@ -703,9 +703,21 @@ protected CompletableFuture
internalSetNamespaceReplicationClusters(List
-
validateClusterForTenantAsync(
-
namespaceName.getTenant(), clusterId));
+.thenCompose(__ ->
getNamespacePoliciesAsync(this.namespaceName)
+
.thenCompose(nsPolicies -> {
+if
(nsPolicies.allowed_clusters.isEmpty()) {
+return
validateClusterForTenantAsync(
+
namespaceName.getTenant(), clusterId);
+}
+if
(!nsPolicies.allowed_clusters.contains(clusterId)) {
Review Comment:
Maybe we can add documentation to make it easier to use?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [fix] [client] Fix resource leak in Pulsar Client since HttpLookupService doesn't get closed [pulsar]
poorbarcode commented on code in PR #22858:
URL: https://github.com/apache/pulsar/pull/22858#discussion_r1634081704
##
pulsar-client/src/main/java/org/apache/pulsar/client/impl/PulsarClientImpl.java:
##
@@ -744,6 +745,21 @@ public void close() throws PulsarClientException {
}
}
+private void closeUrlLookupMap() {
+Map closedUrlLookupServices = new
HashMap(urlLookupMap.size());
+urlLookupMap.entrySet().forEach(e -> {
+try {
+e.getValue().close();
+} catch (Exception ex) {
+log.error("Error closing lookup service {}", e.getKey(), ex);
+}
+closedUrlLookupServices.put(e.getKey(), e.getValue());
+});
+closedUrlLookupServices.entrySet().forEach(e -> {
Review Comment:
```java
1. urlLookupMap.clear();
2. urlLookupMap.putAll(failedLookupMap);
```
If we just look at the method `closeUrlLookupMap `, after `line-1` and
before `line-2`, a command `map.computIfAbsent(k, v)` that using the same key
in the map `failedLookupMap` may have been called, so it is not perfect. The
current implementation is not simple to read, but it guarantees the correct
logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [feat][broker] PIP-321 Introduce allowed-cluster at the namespace level [pulsar]
poorbarcode commented on code in PR #22378:
URL: https://github.com/apache/pulsar/pull/22378#discussion_r1634046614
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java:
##
@@ -703,9 +703,21 @@ protected CompletableFuture
internalSetNamespaceReplicationClusters(List
-
validateClusterForTenantAsync(
-
namespaceName.getTenant(), clusterId));
+.thenCompose(__ ->
getNamespacePoliciesAsync(this.namespaceName)
+
.thenCompose(nsPolicies -> {
+if
(nsPolicies.allowed_clusters.isEmpty()) {
+return
validateClusterForTenantAsync(
+
namespaceName.getTenant(), clusterId);
+}
+if
(!nsPolicies.allowed_clusters.contains(clusterId)) {
Review Comment:
Maybe we should pay attention to how to make it easier to use, do you think
an optional param `--also-set-allowed-clusters` is useful?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[I] DivideByZeroException in ChoosePartition [pulsar-dotpulsar]
smbecker opened a new issue, #224: URL: https://github.com/apache/pulsar-dotpulsar/issues/224 ### Description I cannot replicate it very consistently but it seems to be related to creating a producer and then immediately trying to send on it. In some cases, a `DivideByZeroException` is thrown because `_producerCount` has not been initialized yet. I'm guessing some sort of race condition between [this await](https://github.com/apache/pulsar-dotpulsar/blob/master/src/DotPulsar/Internal/Producer.cs#L205) and `_producerCount` being initialized [here](https://github.com/apache/pulsar-dotpulsar/blob/master/src/DotPulsar/Internal/Producer.cs#L122). Perhaps it is as simple as just changing [this if condition](https://github.com/apache/pulsar-dotpulsar/blob/master/src/DotPulsar/Internal/Producer.cs#L203) to a `while` loop to ensure that `_producerCount` is not 0 before allowing it to proceed. Happy to push a PR if that would be helpful. ### Reproduction Steps Create a producer and immediately attempt to send on it. It doesn't always happen. ### Expected behavior The message sends as expected ### Actual behavior ``` System.DivideByZeroException: Attempted to divide by zero. at DotPulsar.RoundRobinPartitionRouter.ChoosePartition(MessageMetadata messageMetadata, Int32 numberOfPartitions) at DotPulsar.Internal.Producer`1.ChoosePartitions(MessageMetadata metadata, CancellationToken cancellationToken) at DotPulsar.Internal.Producer`1.InternalSend(MessageMetadata metadata, TMessage message, Boolean sendOpCancelable, TaskCompletionSource`1 tcs, Func`2 onMessageSent, Action`1 onFailed, CancellationToken cancellationToken) at DotPulsar.Internal.Producer`1.Send(MessageMetadata metadata, TMessage message, CancellationToken cancellationToken) ### Regression? _No response_ ### Known Workarounds _No response_ ### Configuration _No response_ ### Other information _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [improve] [pip] PIP-355: Enhancing Broker-Level Metrics for Pulsar [pulsar]
dragosvictor commented on code in PR #22778: URL: https://github.com/apache/pulsar/pull/22778#discussion_r1633765907 ## pip/pip-355.md: ## @@ -0,0 +1,39 @@ +# PIP-355: Enhancing Broker-Level Metrics for Pulsar + +# Background Knowledge +Pulsar provides broker-level, namespace-level, and topic-level metrics to monitor and analyze the behavior of the Pulsar service. These metrics are accessible through the Prometheus metrics endpoint. Detailed explanations of all metrics can be found on the Pulsar website: [Pulsar Metrics Reference](https://pulsar.apache.org/docs/3.2.x/reference-metrics/) + +# Motivation +Within Pulsar's current metrics framework, the `pulsar_out_bytes_total` metric is utilized to expose the total bytes dispatched by the broker to consumers. However, there are notable limitations and challenges associated with this metric: +- Inclusion of system subscriptions in the total bytes out, alongside user subscriptions, complicates accurate calculation of user-specific data. +- The granularity of the metric (namespace-level vs. topic-subscription level) impacts the scalability and resource consumption when calculating cluster-level total out bytes. + +# Goals +This proposal aims to address the following objectives: +- Simplify the process of calculating cluster-level total out bytes. +- Enable the calculation of total out bytes dispatched to system subscriptions. + +# High-Level Design +To achieve the outlined goals, the proposal introduces three new broker-level metrics: +- `pulsar_broker_out_bytes_total`: Represents the total out bytes dispatched by the broker to consumers, encompassing both user and system subscriptions. Review Comment: Sounds good to me :+1: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch branch-3.2 updated (613a68e0a85 -> 7914163babb)
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a change to branch branch-3.2 in repository https://gitbox.apache.org/repos/asf/pulsar.git from 613a68e0a85 [improve][ci] Migrate from Gradle Enterprise to Develocity (#22880) new ce9f15f477b [fix][cli] Fix the shell script parameter passthrough syntax (#22867) new 140685c6475 [fix][cli] Fix Pulsar standalone shutdown - bkCluster wasn't closed (#22868) new 9b2b437ffb3 [fix][cli] Fix Pulsar standalone "--wipe-data" (#22885) new 7914163babb [improve] Upgrade IPAddress to 5.5.0 (#22886) The 4 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: bin/bookkeeper | 12 ++--- bin/pulsar | 38 +++--- distribution/server/src/assemble/LICENSE.bin.txt | 2 +- pom.xml| 2 +- .../java/org/apache/pulsar/PulsarStandalone.java | 19 ++- .../org/apache/pulsar/PulsarStandaloneStarter.java | 58 -- .../pulsar/zookeeper/LocalBookkeeperEnsemble.java | 2 + .../org/apache/pulsar/PulsarStandaloneTest.java| 48 -- .../configurations/pulsar_broker_test.conf | 26 +- .../pulsar_broker_test_standalone.conf | 26 +- ...pulsar_broker_test_standalone_with_rocksdb.conf | 26 +- .../configurations/standalone_no_client_auth.conf | 4 +- .../pulsar/metadata/bookkeeper/BKCluster.java | 44 ++-- 13 files changed, 205 insertions(+), 102 deletions(-)
(pulsar) 01/04: [fix][cli] Fix the shell script parameter passthrough syntax (#22867)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit ce9f15f477bcc11a5d421253f9e486fe15fa5c4f
Author: Lari Hotari
AuthorDate: Fri Jun 7 15:25:35 2024 +0300
[fix][cli] Fix the shell script parameter passthrough syntax (#22867)
(cherry picked from commit c81c0f684f8c55c2e39739c6e1de935dff2085d6)
---
bin/bookkeeper | 12 ++--
bin/pulsar | 38 +++---
2 files changed, 25 insertions(+), 25 deletions(-)
diff --git a/bin/bookkeeper b/bin/bookkeeper
index 0cc07dd49ab..13d092f4c99 100755
--- a/bin/bookkeeper
+++ b/bin/bookkeeper
@@ -214,20 +214,20 @@ OPTS="$OPTS $BK_METADATA_OPTIONS"
#Change to BK_HOME to support relative paths
cd "$BK_HOME"
if [ $COMMAND == "bookie" ]; then
-exec $JAVA $OPTS $JMX_ARGS org.apache.bookkeeper.server.Main --conf
$BOOKIE_CONF $@
+exec $JAVA $OPTS $JMX_ARGS org.apache.bookkeeper.server.Main --conf
$BOOKIE_CONF "$@"
elif [ $COMMAND == "autorecovery" ]; then
-exec $JAVA $OPTS $JMX_ARGS
org.apache.bookkeeper.replication.AutoRecoveryMain --conf $BOOKIE_CONF $@
+exec $JAVA $OPTS $JMX_ARGS
org.apache.bookkeeper.replication.AutoRecoveryMain --conf $BOOKIE_CONF "$@"
elif [ $COMMAND == "localbookie" ]; then
NUMBER=$1
shift
-exec $JAVA $OPTS $JMX_ARGS org.apache.bookkeeper.util.LocalBookKeeper
$NUMBER $BOOKIE_CONF $@
+exec $JAVA $OPTS $JMX_ARGS org.apache.bookkeeper.util.LocalBookKeeper
$NUMBER $BOOKIE_CONF "$@"
elif [ $COMMAND == "upgrade" ]; then
-exec $JAVA $OPTS org.apache.bookkeeper.bookie.FileSystemUpgrade --conf
$BOOKIE_CONF $@
+exec $JAVA $OPTS org.apache.bookkeeper.bookie.FileSystemUpgrade --conf
$BOOKIE_CONF "$@"
elif [ $COMMAND == "shell" ]; then
ENTRY_FORMATTER_ARG="-DentryFormatterClass=${ENTRY_FORMATTER_CLASS:-org.apache.bookkeeper.util.StringEntryFormatter}"
-exec $JAVA $OPTS $ENTRY_FORMATTER_ARG
org.apache.bookkeeper.bookie.BookieShell -conf $BOOKIE_CONF $@
+exec $JAVA $OPTS $ENTRY_FORMATTER_ARG
org.apache.bookkeeper.bookie.BookieShell -conf $BOOKIE_CONF "$@"
elif [ $COMMAND == "help" -o $COMMAND == "--help" -o $COMMAND == "-h" ]; then
bookkeeper_help;
else
-exec $JAVA $OPTS $COMMAND $@
+exec $JAVA $OPTS $COMMAND "$@"
fi
diff --git a/bin/pulsar b/bin/pulsar
index ab0029af5b0..f6061601d88 100755
--- a/bin/pulsar
+++ b/bin/pulsar
@@ -329,56 +329,56 @@ fi
cd "$PULSAR_HOME"
if [ $COMMAND == "broker" ]; then
PULSAR_LOG_FILE=${PULSAR_LOG_FILE:-"pulsar-broker.log"}
-exec $JAVA $LOG4J2_SHUTDOWN_HOOK_DISABLED $OPTS
-Dpulsar.log.file=$PULSAR_LOG_FILE org.apache.pulsar.PulsarBrokerStarter
--broker-conf $PULSAR_BROKER_CONF $@
+exec $JAVA $LOG4J2_SHUTDOWN_HOOK_DISABLED $OPTS
-Dpulsar.log.file=$PULSAR_LOG_FILE org.apache.pulsar.PulsarBrokerStarter
--broker-conf $PULSAR_BROKER_CONF "$@"
elif [ $COMMAND == "bookie" ]; then
PULSAR_LOG_FILE=${PULSAR_LOG_FILE:-"bookkeeper.log"}
-exec $JAVA $OPTS -Dpulsar.log.file=$PULSAR_LOG_FILE
org.apache.bookkeeper.server.Main --conf $PULSAR_BOOKKEEPER_CONF $@
+exec $JAVA $OPTS -Dpulsar.log.file=$PULSAR_LOG_FILE
org.apache.bookkeeper.server.Main --conf $PULSAR_BOOKKEEPER_CONF "$@"
elif [ $COMMAND == "zookeeper" ]; then
PULSAR_LOG_FILE=${PULSAR_LOG_FILE:-"zookeeper.log"}
-exec $JAVA ${ZK_OPTS} $OPTS -Dpulsar.log.file=$PULSAR_LOG_FILE
org.apache.zookeeper.server.quorum.QuorumPeerMain $PULSAR_ZK_CONF $@
+exec $JAVA ${ZK_OPTS} $OPTS -Dpulsar.log.file=$PULSAR_LOG_FILE
org.apache.zookeeper.server.quorum.QuorumPeerMain $PULSAR_ZK_CONF "$@"
elif [ $COMMAND == "global-zookeeper" ]; then
PULSAR_LOG_FILE=${PULSAR_LOG_FILE:-"global-zookeeper.log"}
# Allow global ZK to turn into read-only mode when it cannot reach the
quorum
OPTS="${OPTS} ${ZK_OPTS} -Dreadonlymode.enabled=true"
-exec $JAVA $OPTS -Dpulsar.log.file=$PULSAR_LOG_FILE
org.apache.zookeeper.server.quorum.QuorumPeerMain $PULSAR_GLOBAL_ZK_CONF $@
+exec $JAVA $OPTS -Dpulsar.log.file=$PULSAR_LOG_FILE
org.apache.zookeeper.server.quorum.QuorumPeerMain $PULSAR_GLOBAL_ZK_CONF "$@"
elif [ $COMMAND == "configuration-store" ]; then
PULSAR_LOG_FILE=${PULSAR_LOG_FILE:-"configuration-store.log"}
# Allow global ZK to turn into read-only mode when it cannot reach the
quorum
OPTS="${OPTS} ${ZK_OPTS} -Dreadonlymode.enabled=true"
-exec $JAVA $OPTS -Dpulsar.log.file=$PULSAR_LOG_FILE
org.apache.zookeeper.server.quorum.QuorumPeerMain
$PULSAR_CONFIGURATION_STORE_CONF $@
+exec $JAVA $OPTS -Dpulsar.log.file=$PULSAR_LOG_FILE
org.apache.zookeeper.server.quorum.QuorumPeerMain
$PULSAR_CONFIGURATION_STORE_CONF "$@"
elif [ $COMMAND == "proxy" ]; then
PULSAR_LOG_FILE=${PULSAR_LOG_FILE:-"pulsar-proxy.log"}
-exec $JAVA $OPTS -Dpulsar.log.file=$PULSAR_LOG_FILE
org.apache.pulsar.proxy.server.ProxyServiceStarter --config $PULSAR_PROXY_CONF
$@
+exec $JAVA
(pulsar) 02/04: [fix][cli] Fix Pulsar standalone shutdown - bkCluster wasn't closed (#22868)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 140685c64755097528b7c7b5625daa38ba1da408
Author: Lari Hotari
AuthorDate: Fri Jun 7 18:36:52 2024 +0300
[fix][cli] Fix Pulsar standalone shutdown - bkCluster wasn't closed (#22868)
(cherry picked from commit c5cc25ebdc3a32d002b944e77fb59c9ccd1f14c1)
---
.../java/org/apache/pulsar/PulsarStandalone.java | 10
.../org/apache/pulsar/PulsarStandaloneStarter.java | 58 --
.../org/apache/pulsar/PulsarStandaloneTest.java| 48 --
.../configurations/pulsar_broker_test.conf | 26 +-
.../pulsar_broker_test_standalone.conf | 26 +-
...pulsar_broker_test_standalone_with_rocksdb.conf | 26 +-
.../configurations/standalone_no_client_auth.conf | 4 +-
.../pulsar/metadata/bookkeeper/BKCluster.java | 43 ++--
8 files changed, 167 insertions(+), 74 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
index ba136e7c910..a2a101fe394 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
@@ -417,18 +417,22 @@ public class PulsarStandalone implements AutoCloseable {
try {
if (fnWorkerService != null) {
fnWorkerService.stop();
+fnWorkerService = null;
}
if (broker != null) {
broker.close();
+broker = null;
}
if (bkCluster != null) {
bkCluster.close();
+bkCluster = null;
}
if (bkEnsemble != null) {
bkEnsemble.stop();
+bkEnsemble = null;
}
} catch (Exception e) {
log.error("Shutdown failed: {}", e.getMessage(), e);
@@ -493,5 +497,11 @@ public class PulsarStandalone implements AutoCloseable {
ShutdownUtil.triggerImmediateForcefulShutdown(exitCode);
}
+public String getBrokerServiceUrl() {
+return broker.getBrokerServiceUrl();
+}
+public String getWebServiceUrl() {
+return broker.getWebServiceAddress();
+}
}
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandaloneStarter.java
b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandaloneStarter.java
index e3a5e66d4b6..6c14455ab51 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandaloneStarter.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandaloneStarter.java
@@ -21,9 +21,12 @@ package org.apache.pulsar;
import static org.apache.commons.lang3.StringUtils.isBlank;
import com.beust.jcommander.JCommander;
import com.beust.jcommander.Parameter;
+import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Strings;
import java.io.FileInputStream;
import java.util.Arrays;
+import lombok.AccessLevel;
+import lombok.Setter;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.apache.logging.log4j.LogManager;
@@ -38,6 +41,9 @@ public class PulsarStandaloneStarter extends PulsarStandalone
{
@Parameter(names = {"-g", "--generate-docs"}, description = "Generate
docs")
private boolean generateDocs = false;
+private Thread shutdownThread;
+@Setter(AccessLevel.PACKAGE)
+private boolean testMode;
public PulsarStandaloneStarter(String[] args) throws Exception {
@@ -108,30 +114,54 @@ public class PulsarStandaloneStarter extends
PulsarStandalone {
}
}
}
+}
+@Override
+public synchronized void start() throws Exception {
registerShutdownHook();
+super.start();
}
protected void registerShutdownHook() {
-Runtime.getRuntime().addShutdownHook(new Thread(() -> {
+if (shutdownThread != null) {
+throw new IllegalStateException("Shutdown hook already
registered");
+}
+shutdownThread = new Thread(() -> {
try {
-if (fnWorkerService != null) {
-fnWorkerService.stop();
-}
-
-if (broker != null) {
-broker.close();
-}
-
-if (bkEnsemble != null) {
-bkEnsemble.stop();
-}
+doClose(false);
} catch (Exception e) {
log.error("Shutdown failed: {}", e.getMessage(), e);
} finally {
-LogManager.shutdown();
+if (!testMode) {
+LogManager.shutdown();
+}
}
-}));
+});
+
(pulsar) 04/04: [improve] Upgrade IPAddress to 5.5.0 (#22886)
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a commit to branch branch-3.2 in repository https://gitbox.apache.org/repos/asf/pulsar.git commit 7914163babb42baf45e36c5d4350e1f23d1880c5 Author: Matteo Merli AuthorDate: Mon Jun 10 12:39:49 2024 -0700 [improve] Upgrade IPAddress to 5.5.0 (#22886) (cherry picked from commit f17d90e528687fc796cc7e9c5c5b7487a3e3723e) --- distribution/server/src/assemble/LICENSE.bin.txt | 2 +- pom.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/distribution/server/src/assemble/LICENSE.bin.txt b/distribution/server/src/assemble/LICENSE.bin.txt index 2a12383ea65..7f8410b0472 100644 --- a/distribution/server/src/assemble/LICENSE.bin.txt +++ b/distribution/server/src/assemble/LICENSE.bin.txt @@ -501,7 +501,7 @@ The Apache Software License, Version 2.0 - io.etcd-jetcd-core-0.7.7.jar - io.etcd-jetcd-grpc-0.7.7.jar * IPAddress -- com.github.seancfoley-ipaddress-5.3.3.jar +- com.github.seancfoley-ipaddress-5.5.0.jar * RxJava - io.reactivex.rxjava3-rxjava-3.0.1.jar * RoaringBitmap diff --git a/pom.xml b/pom.xml index da220b245a4..7ad9c66db82 100644 --- a/pom.xml +++ b/pom.xml @@ -251,7 +251,7 @@ flexible messaging model and an intuitive client API. 0.7.7 2.0 1.10.12 -5.3.3 +5.5.0 3.4.3 1.5.2-3 2.0.6
(pulsar) 03/04: [fix][cli] Fix Pulsar standalone "--wipe-data" (#22885)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 9b2b437ffb3f0b7546d8b32c1de6c875b3d3db55
Author: Lari Hotari
AuthorDate: Mon Jun 10 19:30:24 2024 +0300
[fix][cli] Fix Pulsar standalone "--wipe-data" (#22885)
(cherry picked from commit f6eceedbded53cded4dd751206ebb51d2867e978)
---
.../src/main/java/org/apache/pulsar/PulsarStandalone.java| 9 -
.../org/apache/pulsar/zookeeper/LocalBookkeeperEnsemble.java | 2 ++
.../java/org/apache/pulsar/metadata/bookkeeper/BKCluster.java| 1 +
3 files changed, 11 insertions(+), 1 deletion(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
index a2a101fe394..360f8caeba6 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
@@ -18,6 +18,7 @@
*/
package org.apache.pulsar;
+import static org.apache.commons.io.FileUtils.cleanDirectory;
import static org.apache.pulsar.common.naming.NamespaceName.SYSTEM_NAMESPACE;
import static
org.apache.pulsar.common.naming.SystemTopicNames.TRANSACTION_COORDINATOR_ASSIGN;
import com.beust.jcommander.Parameter;
@@ -25,6 +26,7 @@ import com.google.common.annotations.VisibleForTesting;
import com.google.common.collect.Sets;
import io.netty.util.internal.PlatformDependent;
import java.io.File;
+import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.List;
@@ -443,7 +445,12 @@ public class PulsarStandalone implements AutoCloseable {
void startBookieWithMetadataStore() throws Exception {
if (StringUtils.isBlank(metadataStoreUrl)){
log.info("Starting BK with RocksDb metadata store");
-metadataStoreUrl = "rocksdb://" +
Paths.get(metadataDir).toAbsolutePath();
+Path metadataDirPath = Paths.get(metadataDir);
+metadataStoreUrl = "rocksdb://" + metadataDirPath.toAbsolutePath();
+if (wipeData && Files.exists(metadataDirPath)) {
+log.info("Wiping RocksDb metadata store at {}",
metadataStoreUrl);
+cleanDirectory(metadataDirPath.toFile());
+}
} else {
log.info("Starting BK with metadata store: {}", metadataStoreUrl);
}
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/zookeeper/LocalBookkeeperEnsemble.java
b/pulsar-broker/src/main/java/org/apache/pulsar/zookeeper/LocalBookkeeperEnsemble.java
index 63d146a3a15..4c8d2dbbfa7 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/zookeeper/LocalBookkeeperEnsemble.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/zookeeper/LocalBookkeeperEnsemble.java
@@ -194,6 +194,7 @@ public class LocalBookkeeperEnsemble {
: createTempDirectory("zktest");
if (this.clearOldData) {
+LOG.info("Wiping Zookeeper data directory at {}",
zkDataDir.getAbsolutePath());
cleanDirectory(zkDataDir);
}
@@ -291,6 +292,7 @@ public class LocalBookkeeperEnsemble {
: createTempDirectory("bk" + i + "test");
if (this.clearOldData) {
+LOG.info("Wiping Bookie data directory at {}",
bkDataDir.getAbsolutePath());
cleanDirectory(bkDataDir);
}
diff --git
a/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/BKCluster.java
b/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/BKCluster.java
index 8d3a90239ef..fe2b981ffe9 100644
---
a/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/BKCluster.java
+++
b/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/BKCluster.java
@@ -232,6 +232,7 @@ public class BKCluster implements AutoCloseable {
}
if (clusterConf.clearOldData && dataDir.exists()) {
+log.info("Wiping Bookie data directory at {}",
dataDir.getAbsolutePath());
cleanDirectory(dataDir);
}
(pulsar) branch branch-3.0 updated: [improve] Upgrade IPAddress to 5.5.0 (#22886)
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new caf08c26f58 [improve] Upgrade IPAddress to 5.5.0 (#22886) caf08c26f58 is described below commit caf08c26f586eb1e09a14755916a769b60474008 Author: Matteo Merli AuthorDate: Mon Jun 10 12:39:49 2024 -0700 [improve] Upgrade IPAddress to 5.5.0 (#22886) (cherry picked from commit f17d90e528687fc796cc7e9c5c5b7487a3e3723e) --- distribution/server/src/assemble/LICENSE.bin.txt | 2 +- pom.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/distribution/server/src/assemble/LICENSE.bin.txt b/distribution/server/src/assemble/LICENSE.bin.txt index 05859c8f720..4d050985f33 100644 --- a/distribution/server/src/assemble/LICENSE.bin.txt +++ b/distribution/server/src/assemble/LICENSE.bin.txt @@ -501,7 +501,7 @@ The Apache Software License, Version 2.0 - io.etcd-jetcd-core-0.7.7.jar - io.etcd-jetcd-grpc-0.7.7.jar * IPAddress -- com.github.seancfoley-ipaddress-5.3.3.jar +- com.github.seancfoley-ipaddress-5.5.0.jar * RxJava - io.reactivex.rxjava3-rxjava-3.0.1.jar * RoaringBitmap diff --git a/pom.xml b/pom.xml index 51250ec6b57..9ac151670f3 100644 --- a/pom.xml +++ b/pom.xml @@ -251,7 +251,7 @@ flexible messaging model and an intuitive client API. 0.7.7 2.0 1.10.12 -5.3.3 +5.5.0 3.4.3 1.5.2-3 2.0.6
(pulsar) branch branch-3.3 updated: [improve] Upgrade IPAddress to 5.5.0 (#22886)
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a commit to branch branch-3.3 in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/branch-3.3 by this push: new b2422c0b18c [improve] Upgrade IPAddress to 5.5.0 (#22886) b2422c0b18c is described below commit b2422c0b18c110cf501613a0efbadecbd31ce944 Author: Matteo Merli AuthorDate: Mon Jun 10 12:39:49 2024 -0700 [improve] Upgrade IPAddress to 5.5.0 (#22886) (cherry picked from commit f17d90e528687fc796cc7e9c5c5b7487a3e3723e) --- distribution/server/src/assemble/LICENSE.bin.txt | 2 +- pom.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/distribution/server/src/assemble/LICENSE.bin.txt b/distribution/server/src/assemble/LICENSE.bin.txt index 4101804a792..dd033ad05ac 100644 --- a/distribution/server/src/assemble/LICENSE.bin.txt +++ b/distribution/server/src/assemble/LICENSE.bin.txt @@ -516,7 +516,7 @@ The Apache Software License, Version 2.0 - io.etcd-jetcd-core-0.7.7.jar - io.etcd-jetcd-grpc-0.7.7.jar * IPAddress -- com.github.seancfoley-ipaddress-5.3.3.jar +- com.github.seancfoley-ipaddress-5.5.0.jar * RxJava - io.reactivex.rxjava3-rxjava-3.0.1.jar * RoaringBitmap diff --git a/pom.xml b/pom.xml index 5883d20b9da..3d9357c149a 100644 --- a/pom.xml +++ b/pom.xml @@ -252,7 +252,7 @@ flexible messaging model and an intuitive client API. 0.2.0 2.0 1.10.12 -5.3.3 +5.5.0 3.4.3 1.5.2-3 2.0.6
(pulsar) branch master updated: [improve] Upgrade IPAddress to 5.5.0 (#22886)
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new f17d90e5286 [improve] Upgrade IPAddress to 5.5.0 (#22886) f17d90e5286 is described below commit f17d90e528687fc796cc7e9c5c5b7487a3e3723e Author: Matteo Merli AuthorDate: Mon Jun 10 12:39:49 2024 -0700 [improve] Upgrade IPAddress to 5.5.0 (#22886) --- distribution/server/src/assemble/LICENSE.bin.txt | 2 +- pom.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/distribution/server/src/assemble/LICENSE.bin.txt b/distribution/server/src/assemble/LICENSE.bin.txt index 25b6787d420..6769df39037 100644 --- a/distribution/server/src/assemble/LICENSE.bin.txt +++ b/distribution/server/src/assemble/LICENSE.bin.txt @@ -516,7 +516,7 @@ The Apache Software License, Version 2.0 - io.etcd-jetcd-core-0.7.7.jar - io.etcd-jetcd-grpc-0.7.7.jar * IPAddress -- com.github.seancfoley-ipaddress-5.3.3.jar +- com.github.seancfoley-ipaddress-5.5.0.jar * RxJava - io.reactivex.rxjava3-rxjava-3.0.1.jar * RoaringBitmap diff --git a/pom.xml b/pom.xml index 6e32359f326..1514b7da13a 100644 --- a/pom.xml +++ b/pom.xml @@ -255,7 +255,7 @@ flexible messaging model and an intuitive client API. 0.3.0 2.0 1.10.12 -5.3.3 +5.5.0 3.4.3 1.5.2-3 2.0.6
Re: [PR] [improve] Upgrade IPAddress to 5.5.0 [pulsar]
lhotari merged PR #22886: URL: https://github.com/apache/pulsar/pull/22886 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch branch-3.3 updated: [fix][cli] Fix Pulsar standalone "--wipe-data" (#22885)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new 4bc4171db81 [fix][cli] Fix Pulsar standalone "--wipe-data" (#22885)
4bc4171db81 is described below
commit 4bc4171db81397cecba31d831c20ee16f72365ed
Author: Lari Hotari
AuthorDate: Mon Jun 10 19:30:24 2024 +0300
[fix][cli] Fix Pulsar standalone "--wipe-data" (#22885)
(cherry picked from commit f6eceedbded53cded4dd751206ebb51d2867e978)
---
.../src/main/java/org/apache/pulsar/PulsarStandalone.java| 9 -
.../org/apache/pulsar/zookeeper/LocalBookkeeperEnsemble.java | 2 ++
.../java/org/apache/pulsar/metadata/bookkeeper/BKCluster.java| 1 +
3 files changed, 11 insertions(+), 1 deletion(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
index 7f80aa29f53..d0118b06e7c 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
@@ -18,12 +18,14 @@
*/
package org.apache.pulsar;
+import static org.apache.commons.io.FileUtils.cleanDirectory;
import static org.apache.pulsar.common.naming.NamespaceName.SYSTEM_NAMESPACE;
import static
org.apache.pulsar.common.naming.SystemTopicNames.TRANSACTION_COORDINATOR_ASSIGN;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.collect.Sets;
import io.netty.util.internal.PlatformDependent;
import java.io.File;
+import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.List;
@@ -446,7 +448,12 @@ public class PulsarStandalone implements AutoCloseable {
void startBookieWithMetadataStore() throws Exception {
if (StringUtils.isBlank(metadataStoreUrl)){
log.info("Starting BK with RocksDb metadata store");
-metadataStoreUrl = "rocksdb://" +
Paths.get(metadataDir).toAbsolutePath();
+Path metadataDirPath = Paths.get(metadataDir);
+metadataStoreUrl = "rocksdb://" + metadataDirPath.toAbsolutePath();
+if (wipeData && Files.exists(metadataDirPath)) {
+log.info("Wiping RocksDb metadata store at {}",
metadataStoreUrl);
+cleanDirectory(metadataDirPath.toFile());
+}
} else {
log.info("Starting BK with metadata store: {}", metadataStoreUrl);
}
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/zookeeper/LocalBookkeeperEnsemble.java
b/pulsar-broker/src/main/java/org/apache/pulsar/zookeeper/LocalBookkeeperEnsemble.java
index e8a503c46e0..cf1a30951eb 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/zookeeper/LocalBookkeeperEnsemble.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/zookeeper/LocalBookkeeperEnsemble.java
@@ -194,6 +194,7 @@ public class LocalBookkeeperEnsemble {
: createTempDirectory("zktest");
if (this.clearOldData) {
+LOG.info("Wiping Zookeeper data directory at {}",
zkDataDir.getAbsolutePath());
cleanDirectory(zkDataDir);
}
@@ -291,6 +292,7 @@ public class LocalBookkeeperEnsemble {
: createTempDirectory("bk" + i + "test");
if (this.clearOldData) {
+LOG.info("Wiping Bookie data directory at {}",
bkDataDir.getAbsolutePath());
cleanDirectory(bkDataDir);
}
diff --git
a/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/BKCluster.java
b/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/BKCluster.java
index 8d3a90239ef..fe2b981ffe9 100644
---
a/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/BKCluster.java
+++
b/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/BKCluster.java
@@ -232,6 +232,7 @@ public class BKCluster implements AutoCloseable {
}
if (clusterConf.clearOldData && dataDir.exists()) {
+log.info("Wiping Bookie data directory at {}",
dataDir.getAbsolutePath());
cleanDirectory(dataDir);
}
(pulsar) 03/04: [fix][cli] Fix Pulsar standalone shutdown - bkCluster wasn't closed (#22868)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit ebb4282949e53ecf2229525d840143830beec89b
Author: Lari Hotari
AuthorDate: Fri Jun 7 18:36:52 2024 +0300
[fix][cli] Fix Pulsar standalone shutdown - bkCluster wasn't closed (#22868)
(cherry picked from commit c5cc25ebdc3a32d002b944e77fb59c9ccd1f14c1)
---
.../java/org/apache/pulsar/PulsarStandalone.java | 10
.../org/apache/pulsar/PulsarStandaloneStarter.java | 58 --
.../org/apache/pulsar/PulsarStandaloneTest.java| 48 --
.../configurations/pulsar_broker_test.conf | 26 +-
.../pulsar_broker_test_standalone.conf | 26 +-
...pulsar_broker_test_standalone_with_rocksdb.conf | 26 +-
.../configurations/standalone_no_client_auth.conf | 4 +-
.../pulsar/metadata/bookkeeper/BKCluster.java | 43 ++--
8 files changed, 167 insertions(+), 74 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
index ba136e7c910..a2a101fe394 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
@@ -417,18 +417,22 @@ public class PulsarStandalone implements AutoCloseable {
try {
if (fnWorkerService != null) {
fnWorkerService.stop();
+fnWorkerService = null;
}
if (broker != null) {
broker.close();
+broker = null;
}
if (bkCluster != null) {
bkCluster.close();
+bkCluster = null;
}
if (bkEnsemble != null) {
bkEnsemble.stop();
+bkEnsemble = null;
}
} catch (Exception e) {
log.error("Shutdown failed: {}", e.getMessage(), e);
@@ -493,5 +497,11 @@ public class PulsarStandalone implements AutoCloseable {
ShutdownUtil.triggerImmediateForcefulShutdown(exitCode);
}
+public String getBrokerServiceUrl() {
+return broker.getBrokerServiceUrl();
+}
+public String getWebServiceUrl() {
+return broker.getWebServiceAddress();
+}
}
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandaloneStarter.java
b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandaloneStarter.java
index 33dd58deac0..b84b829d9b1 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandaloneStarter.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandaloneStarter.java
@@ -21,9 +21,12 @@ package org.apache.pulsar;
import static org.apache.commons.lang3.StringUtils.isBlank;
import com.beust.jcommander.JCommander;
import com.beust.jcommander.Parameter;
+import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Strings;
import java.io.FileInputStream;
import java.util.Arrays;
+import lombok.AccessLevel;
+import lombok.Setter;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.apache.logging.log4j.LogManager;
@@ -38,6 +41,9 @@ public class PulsarStandaloneStarter extends PulsarStandalone
{
@Parameter(names = {"-g", "--generate-docs"}, description = "Generate
docs")
private boolean generateDocs = false;
+private Thread shutdownThread;
+@Setter(AccessLevel.PACKAGE)
+private boolean testMode;
public PulsarStandaloneStarter(String[] args) throws Exception {
@@ -108,26 +114,50 @@ public class PulsarStandaloneStarter extends
PulsarStandalone {
}
}
}
+}
-Runtime.getRuntime().addShutdownHook(new Thread(() -> {
+@Override
+public synchronized void start() throws Exception {
+super.start();
+if (shutdownThread != null) {
+throw new IllegalStateException("Shutdown hook already
registered");
+}
+shutdownThread = new Thread(() -> {
try {
-if (fnWorkerService != null) {
-fnWorkerService.stop();
-}
-
-if (broker != null) {
-broker.close();
-}
-
-if (bkEnsemble != null) {
-bkEnsemble.stop();
-}
+doClose(false);
} catch (Exception e) {
log.error("Shutdown failed: {}", e.getMessage(), e);
} finally {
-LogManager.shutdown();
+if (!testMode) {
+LogManager.shutdown();
+}
}
-}));
+});
+Runtime.getRuntime().addShutdownHook(shutdownThread);
+}
+
+// simulate running the shutdown hook,
(pulsar) branch branch-3.0 updated (477499d4fab -> d9928ef944f)
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/pulsar.git from 477499d4fab [improve][ci] Migrate from Gradle Enterprise to Develocity (#22880) new efa5e8b0401 [fix] Remove blocking calls from BookieRackAffinityMapping (#22846) new a7cb5ec9c5f [fix][cli] Fix the shell script parameter passthrough syntax (#22867) new ebb4282949e [fix][cli] Fix Pulsar standalone shutdown - bkCluster wasn't closed (#22868) new d9928ef944f [fix][cli] Fix Pulsar standalone "--wipe-data" (#22885) The 4 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: bin/bookkeeper | 12 ++--- bin/pulsar | 38 +++--- .../rackawareness/BookieRackAffinityMapping.java | 44 +--- .../IsolatedBookieEnsemblePlacementPolicy.java | 2 +- .../java/org/apache/pulsar/PulsarStandalone.java | 19 ++- .../org/apache/pulsar/PulsarStandaloneStarter.java | 58 -- .../pulsar/zookeeper/LocalBookkeeperEnsemble.java | 2 + .../org/apache/pulsar/PulsarStandaloneTest.java| 48 -- .../configurations/pulsar_broker_test.conf | 26 +- .../pulsar_broker_test_standalone.conf | 26 +- ...pulsar_broker_test_standalone_with_rocksdb.conf | 26 +- .../configurations/standalone_no_client_auth.conf | 4 +- .../pulsar/metadata/bookkeeper/BKCluster.java | 44 ++-- 13 files changed, 231 insertions(+), 118 deletions(-)
(pulsar) 01/04: [fix] Remove blocking calls from BookieRackAffinityMapping (#22846)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit efa5e8b04018356447ec1744c6e083430e8e1f05
Author: Matteo Merli
AuthorDate: Wed Jun 5 10:49:00 2024 -0700
[fix] Remove blocking calls from BookieRackAffinityMapping (#22846)
(cherry picked from commit aece67e35ecec4a9d90a951b78cfc89ca6395054)
---
.../rackawareness/BookieRackAffinityMapping.java | 44 +-
.../IsolatedBookieEnsemblePlacementPolicy.java | 2 +-
2 files changed, 28 insertions(+), 18 deletions(-)
diff --git
a/pulsar-broker-common/src/main/java/org/apache/pulsar/bookie/rackawareness/BookieRackAffinityMapping.java
b/pulsar-broker-common/src/main/java/org/apache/pulsar/bookie/rackawareness/BookieRackAffinityMapping.java
index 983822f2294..4a5ff746f40 100644
---
a/pulsar-broker-common/src/main/java/org/apache/pulsar/bookie/rackawareness/BookieRackAffinityMapping.java
+++
b/pulsar-broker-common/src/main/java/org/apache/pulsar/bookie/rackawareness/BookieRackAffinityMapping.java
@@ -70,7 +70,7 @@ public class BookieRackAffinityMapping extends
AbstractDNSToSwitchMapping
private BookiesRackConfiguration racksWithHost = new
BookiesRackConfiguration();
private Map bookieInfoMap = new HashMap<>();
-public static MetadataStore createMetadataStore(Configuration conf) throws
MetadataException {
+static MetadataStore getMetadataStore(Configuration conf) throws
MetadataException {
MetadataStore store;
Object storeProperty = conf.getProperty(METADATA_STORE_INSTANCE);
if (storeProperty != null) {
@@ -116,12 +116,20 @@ public class BookieRackAffinityMapping extends
AbstractDNSToSwitchMapping
super.setConf(conf);
MetadataStore store;
try {
-store = createMetadataStore(conf);
-bookieMappingCache =
store.getMetadataCache(BookiesRackConfiguration.class);
-store.registerListener(this::handleUpdates);
-racksWithHost = bookieMappingCache.get(BOOKIE_INFO_ROOT_PATH).get()
-.orElseGet(BookiesRackConfiguration::new);
-for (Map bookieMapping :
racksWithHost.values()) {
+store = getMetadataStore(conf);
+} catch (MetadataException e) {
+throw new RuntimeException(METADATA_STORE_INSTANCE + " failed to
init BookieId list");
+}
+
+bookieMappingCache =
store.getMetadataCache(BookiesRackConfiguration.class);
+store.registerListener(this::handleUpdates);
+
+try {
+var racksWithHost = bookieMappingCache.get(BOOKIE_INFO_ROOT_PATH)
+.thenApply(optRes ->
optRes.orElseGet(BookiesRackConfiguration::new))
+.get();
+
+for (var bookieMapping : racksWithHost.values()) {
for (String address : bookieMapping.keySet()) {
bookieAddressListLastTime.add(BookieId.parse(address));
}
@@ -131,10 +139,12 @@ public class BookieRackAffinityMapping extends
AbstractDNSToSwitchMapping
}
}
updateRacksWithHost(racksWithHost);
-watchAvailableBookies();
-} catch (InterruptedException | ExecutionException | MetadataException
e) {
-throw new RuntimeException(METADATA_STORE_INSTANCE + " failed to
init BookieId list");
+} catch (ExecutionException | InterruptedException e) {
+LOG.error("Failed to update rack info. ", e);
+throw new RuntimeException(e);
}
+
+watchAvailableBookies();
}
private void watchAvailableBookies() {
@@ -145,13 +155,13 @@ public class BookieRackAffinityMapping extends
AbstractDNSToSwitchMapping
field.setAccessible(true);
RegistrationClient registrationClient = (RegistrationClient)
field.get(bookieAddressResolver);
registrationClient.watchWritableBookies(versioned -> {
-try {
-racksWithHost =
bookieMappingCache.get(BOOKIE_INFO_ROOT_PATH).get()
-.orElseGet(BookiesRackConfiguration::new);
-updateRacksWithHost(racksWithHost);
-} catch (InterruptedException | ExecutionException e) {
-LOG.error("Failed to update rack info. ", e);
-}
+bookieMappingCache.get(BOOKIE_INFO_ROOT_PATH)
+.thenApply(optRes ->
optRes.orElseGet(BookiesRackConfiguration::new))
+.thenAccept(this::updateRacksWithHost)
+.exceptionally(ex -> {
+LOG.error("Failed to update rack info. ", ex);
+return null;
+});
});
} catch (NoSuchFieldException |
(pulsar) 04/04: [fix][cli] Fix Pulsar standalone "--wipe-data" (#22885)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit d9928ef944f5b2ac8eb2e817704a7d3b69e11d5e
Author: Lari Hotari
AuthorDate: Mon Jun 10 19:30:24 2024 +0300
[fix][cli] Fix Pulsar standalone "--wipe-data" (#22885)
(cherry picked from commit f6eceedbded53cded4dd751206ebb51d2867e978)
---
.../src/main/java/org/apache/pulsar/PulsarStandalone.java| 9 -
.../org/apache/pulsar/zookeeper/LocalBookkeeperEnsemble.java | 2 ++
.../java/org/apache/pulsar/metadata/bookkeeper/BKCluster.java| 1 +
3 files changed, 11 insertions(+), 1 deletion(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
index a2a101fe394..360f8caeba6 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
@@ -18,6 +18,7 @@
*/
package org.apache.pulsar;
+import static org.apache.commons.io.FileUtils.cleanDirectory;
import static org.apache.pulsar.common.naming.NamespaceName.SYSTEM_NAMESPACE;
import static
org.apache.pulsar.common.naming.SystemTopicNames.TRANSACTION_COORDINATOR_ASSIGN;
import com.beust.jcommander.Parameter;
@@ -25,6 +26,7 @@ import com.google.common.annotations.VisibleForTesting;
import com.google.common.collect.Sets;
import io.netty.util.internal.PlatformDependent;
import java.io.File;
+import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.List;
@@ -443,7 +445,12 @@ public class PulsarStandalone implements AutoCloseable {
void startBookieWithMetadataStore() throws Exception {
if (StringUtils.isBlank(metadataStoreUrl)){
log.info("Starting BK with RocksDb metadata store");
-metadataStoreUrl = "rocksdb://" +
Paths.get(metadataDir).toAbsolutePath();
+Path metadataDirPath = Paths.get(metadataDir);
+metadataStoreUrl = "rocksdb://" + metadataDirPath.toAbsolutePath();
+if (wipeData && Files.exists(metadataDirPath)) {
+log.info("Wiping RocksDb metadata store at {}",
metadataStoreUrl);
+cleanDirectory(metadataDirPath.toFile());
+}
} else {
log.info("Starting BK with metadata store: {}", metadataStoreUrl);
}
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/zookeeper/LocalBookkeeperEnsemble.java
b/pulsar-broker/src/main/java/org/apache/pulsar/zookeeper/LocalBookkeeperEnsemble.java
index 63d146a3a15..4c8d2dbbfa7 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/zookeeper/LocalBookkeeperEnsemble.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/zookeeper/LocalBookkeeperEnsemble.java
@@ -194,6 +194,7 @@ public class LocalBookkeeperEnsemble {
: createTempDirectory("zktest");
if (this.clearOldData) {
+LOG.info("Wiping Zookeeper data directory at {}",
zkDataDir.getAbsolutePath());
cleanDirectory(zkDataDir);
}
@@ -291,6 +292,7 @@ public class LocalBookkeeperEnsemble {
: createTempDirectory("bk" + i + "test");
if (this.clearOldData) {
+LOG.info("Wiping Bookie data directory at {}",
bkDataDir.getAbsolutePath());
cleanDirectory(bkDataDir);
}
diff --git
a/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/BKCluster.java
b/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/BKCluster.java
index 8d3a90239ef..fe2b981ffe9 100644
---
a/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/BKCluster.java
+++
b/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/BKCluster.java
@@ -232,6 +232,7 @@ public class BKCluster implements AutoCloseable {
}
if (clusterConf.clearOldData && dataDir.exists()) {
+log.info("Wiping Bookie data directory at {}",
dataDir.getAbsolutePath());
cleanDirectory(dataDir);
}
(pulsar) 02/04: [fix][cli] Fix the shell script parameter passthrough syntax (#22867)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit a7cb5ec9c5f9238cbb8c1a9ecdd13f32636c4da5
Author: Lari Hotari
AuthorDate: Fri Jun 7 15:25:35 2024 +0300
[fix][cli] Fix the shell script parameter passthrough syntax (#22867)
(cherry picked from commit c81c0f684f8c55c2e39739c6e1de935dff2085d6)
---
bin/bookkeeper | 12 ++--
bin/pulsar | 38 +++---
2 files changed, 25 insertions(+), 25 deletions(-)
diff --git a/bin/bookkeeper b/bin/bookkeeper
index 0cc07dd49ab..13d092f4c99 100755
--- a/bin/bookkeeper
+++ b/bin/bookkeeper
@@ -214,20 +214,20 @@ OPTS="$OPTS $BK_METADATA_OPTIONS"
#Change to BK_HOME to support relative paths
cd "$BK_HOME"
if [ $COMMAND == "bookie" ]; then
-exec $JAVA $OPTS $JMX_ARGS org.apache.bookkeeper.server.Main --conf
$BOOKIE_CONF $@
+exec $JAVA $OPTS $JMX_ARGS org.apache.bookkeeper.server.Main --conf
$BOOKIE_CONF "$@"
elif [ $COMMAND == "autorecovery" ]; then
-exec $JAVA $OPTS $JMX_ARGS
org.apache.bookkeeper.replication.AutoRecoveryMain --conf $BOOKIE_CONF $@
+exec $JAVA $OPTS $JMX_ARGS
org.apache.bookkeeper.replication.AutoRecoveryMain --conf $BOOKIE_CONF "$@"
elif [ $COMMAND == "localbookie" ]; then
NUMBER=$1
shift
-exec $JAVA $OPTS $JMX_ARGS org.apache.bookkeeper.util.LocalBookKeeper
$NUMBER $BOOKIE_CONF $@
+exec $JAVA $OPTS $JMX_ARGS org.apache.bookkeeper.util.LocalBookKeeper
$NUMBER $BOOKIE_CONF "$@"
elif [ $COMMAND == "upgrade" ]; then
-exec $JAVA $OPTS org.apache.bookkeeper.bookie.FileSystemUpgrade --conf
$BOOKIE_CONF $@
+exec $JAVA $OPTS org.apache.bookkeeper.bookie.FileSystemUpgrade --conf
$BOOKIE_CONF "$@"
elif [ $COMMAND == "shell" ]; then
ENTRY_FORMATTER_ARG="-DentryFormatterClass=${ENTRY_FORMATTER_CLASS:-org.apache.bookkeeper.util.StringEntryFormatter}"
-exec $JAVA $OPTS $ENTRY_FORMATTER_ARG
org.apache.bookkeeper.bookie.BookieShell -conf $BOOKIE_CONF $@
+exec $JAVA $OPTS $ENTRY_FORMATTER_ARG
org.apache.bookkeeper.bookie.BookieShell -conf $BOOKIE_CONF "$@"
elif [ $COMMAND == "help" -o $COMMAND == "--help" -o $COMMAND == "-h" ]; then
bookkeeper_help;
else
-exec $JAVA $OPTS $COMMAND $@
+exec $JAVA $OPTS $COMMAND "$@"
fi
diff --git a/bin/pulsar b/bin/pulsar
index 20ed1f7f22b..272ed9b8225 100755
--- a/bin/pulsar
+++ b/bin/pulsar
@@ -364,52 +364,52 @@ fi
cd "$PULSAR_HOME"
if [ $COMMAND == "broker" ]; then
PULSAR_LOG_FILE=${PULSAR_LOG_FILE:-"pulsar-broker.log"}
-exec $JAVA $LOG4J2_SHUTDOWN_HOOK_DISABLED $OPTS
-Dpulsar.log.file=$PULSAR_LOG_FILE org.apache.pulsar.PulsarBrokerStarter
--broker-conf $PULSAR_BROKER_CONF $@
+exec $JAVA $LOG4J2_SHUTDOWN_HOOK_DISABLED $OPTS
-Dpulsar.log.file=$PULSAR_LOG_FILE org.apache.pulsar.PulsarBrokerStarter
--broker-conf $PULSAR_BROKER_CONF "$@"
elif [ $COMMAND == "bookie" ]; then
PULSAR_LOG_FILE=${PULSAR_LOG_FILE:-"bookkeeper.log"}
-exec $JAVA $OPTS -Dpulsar.log.file=$PULSAR_LOG_FILE
org.apache.bookkeeper.server.Main --conf $PULSAR_BOOKKEEPER_CONF $@
+exec $JAVA $OPTS -Dpulsar.log.file=$PULSAR_LOG_FILE
org.apache.bookkeeper.server.Main --conf $PULSAR_BOOKKEEPER_CONF "$@"
elif [ $COMMAND == "zookeeper" ]; then
PULSAR_LOG_FILE=${PULSAR_LOG_FILE:-"zookeeper.log"}
-exec $JAVA ${ZK_OPTS} $OPTS -Dpulsar.log.file=$PULSAR_LOG_FILE
org.apache.zookeeper.server.quorum.QuorumPeerMain $PULSAR_ZK_CONF $@
+exec $JAVA ${ZK_OPTS} $OPTS -Dpulsar.log.file=$PULSAR_LOG_FILE
org.apache.zookeeper.server.quorum.QuorumPeerMain $PULSAR_ZK_CONF "$@"
elif [ $COMMAND == "global-zookeeper" ]; then
PULSAR_LOG_FILE=${PULSAR_LOG_FILE:-"global-zookeeper.log"}
# Allow global ZK to turn into read-only mode when it cannot reach the
quorum
OPTS="${OPTS} ${ZK_OPTS} -Dreadonlymode.enabled=true"
-exec $JAVA $OPTS -Dpulsar.log.file=$PULSAR_LOG_FILE
org.apache.zookeeper.server.quorum.QuorumPeerMain $PULSAR_GLOBAL_ZK_CONF $@
+exec $JAVA $OPTS -Dpulsar.log.file=$PULSAR_LOG_FILE
org.apache.zookeeper.server.quorum.QuorumPeerMain $PULSAR_GLOBAL_ZK_CONF "$@"
elif [ $COMMAND == "configuration-store" ]; then
PULSAR_LOG_FILE=${PULSAR_LOG_FILE:-"configuration-store.log"}
# Allow global ZK to turn into read-only mode when it cannot reach the
quorum
OPTS="${OPTS} ${ZK_OPTS} -Dreadonlymode.enabled=true"
-exec $JAVA $OPTS -Dpulsar.log.file=$PULSAR_LOG_FILE
org.apache.zookeeper.server.quorum.QuorumPeerMain
$PULSAR_CONFIGURATION_STORE_CONF $@
+exec $JAVA $OPTS -Dpulsar.log.file=$PULSAR_LOG_FILE
org.apache.zookeeper.server.quorum.QuorumPeerMain
$PULSAR_CONFIGURATION_STORE_CONF "$@"
elif [ $COMMAND == "proxy" ]; then
PULSAR_LOG_FILE=${PULSAR_LOG_FILE:-"pulsar-proxy.log"}
-exec $JAVA $OPTS -Dpulsar.log.file=$PULSAR_LOG_FILE
org.apache.pulsar.proxy.server.ProxyServiceStarter --config $PULSAR_PROXY_CONF
$@
+exec $JAVA
Re: [PR] [improve] Upgrade IPAddress to 5.5.0 [pulsar]
codecov-commenter commented on PR #22886: URL: https://github.com/apache/pulsar/pull/22886#issuecomment-2159126629 ## [Codecov](https://app.codecov.io/gh/apache/pulsar/pull/22886?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report All modified and coverable lines are covered by tests :white_check_mark: > Project coverage is 73.21%. Comparing base [(`bbc6224`)](https://app.codecov.io/gh/apache/pulsar/commit/bbc62245c5ddba1de4b1e7cee4ab49334bc36277?dropdown=coverage=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) to head [(`aa8e8d4`)](https://app.codecov.io/gh/apache/pulsar/commit/aa8e8d49633f30547eef1882b6af05cebaa0628a?dropdown=coverage=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache). > Report is 368 commits behind head on master. Additional details and impacted files [](https://app.codecov.io/gh/apache/pulsar/pull/22886?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) ```diff @@ Coverage Diff @@ ## master #22886 +/- ## - Coverage 73.57% 73.21% -0.37% - Complexity3262432985 +361 Files 1877 1891 +14 Lines139502 141943+2441 Branches 1529915566 +267 + Hits 102638 103921+1283 - Misses289082+1091 - Partials 7956 8023 +67 ``` | [Flag](https://app.codecov.io/gh/apache/pulsar/pull/22886/flags?src=pr=flags_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Coverage Δ | | |---|---|---| | [inttests](https://app.codecov.io/gh/apache/pulsar/pull/22886/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `27.21% <ø> (+2.63%)` | :arrow_up: | | [systests](https://app.codecov.io/gh/apache/pulsar/pull/22886/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `24.66% <ø> (+0.34%)` | :arrow_up: | | [unittests](https://app.codecov.io/gh/apache/pulsar/pull/22886/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `72.24% <ø> (-0.60%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more. [see 392 files with indirect coverage changes](https://app.codecov.io/gh/apache/pulsar/pull/22886/indirect-changes?src=pr=tree-more_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar-site) branch main updated: Rebuild: BUILD_ALL_VERSION=1
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/pulsar-site.git The following commit(s) were added to refs/heads/main by this push: new a31dda2161bc Rebuild: BUILD_ALL_VERSION=1 a31dda2161bc is described below commit a31dda2161bc9ca7c8ecb59b65673386e3c80b53 Author: Lari Hotari AuthorDate: Mon Jun 10 22:03:37 2024 +0300 Rebuild: BUILD_ALL_VERSION=1
(pulsar-site) branch main updated: Pass -Dlocale=en_US when to mvn command line when generating javadoc
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/pulsar-site.git The following commit(s) were added to refs/heads/main by this push: new cfc7e9d13739 Pass -Dlocale=en_US when to mvn command line when generating javadoc cfc7e9d13739 is described below commit cfc7e9d13739a06d8253b42a393d2b21eb1f6180 Author: Lari Hotari AuthorDate: Mon Jun 10 21:52:33 2024 +0300 Pass -Dlocale=en_US when to mvn command line when generating javadoc --- tools/pytools/lib/execute/javadoc_generator.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tools/pytools/lib/execute/javadoc_generator.py b/tools/pytools/lib/execute/javadoc_generator.py index ab7253a8495c..12942bc4e2b5 100644 --- a/tools/pytools/lib/execute/javadoc_generator.py +++ b/tools/pytools/lib/execute/javadoc_generator.py @@ -48,15 +48,15 @@ def execute(version: str): # client dst = site_path() / 'static' / 'api' / 'client' / v -run(mvn, '-pl', 'pulsar-client-api', 'javadoc:javadoc', cwd=src) +run(mvn, '-Dlocale=en_US', '-pl', 'pulsar-client-api', 'javadoc:javadoc', cwd=src) shutil.copytree(src / 'pulsar-client-api' / 'target' / 'site' / 'apidocs', dst) # admin dst = site_path() / 'static' / 'api' / 'admin' / v -run(mvn, '-pl', 'pulsar-client-admin-api', 'javadoc:javadoc', cwd=src) +run(mvn, '-Dlocale=en_US', '-pl', 'pulsar-client-admin-api', 'javadoc:javadoc', cwd=src) shutil.copytree(src / 'pulsar-client-admin-api' / 'target' / 'site' / 'apidocs', dst) # function dst = site_path() / 'static' / 'api' / 'pulsar-functions' / v -run(mvn, '-pl', 'api-java', 'javadoc:javadoc', cwd=(src / 'pulsar-functions')) +run(mvn, '-Dlocale=en_US', '-pl', 'api-java', 'javadoc:javadoc', cwd=(src / 'pulsar-functions')) shutil.copytree(src / 'pulsar-functions' / 'api-java' / 'target' / 'site' / 'apidocs', dst)
[I] [Bug] [docs] Pulsar 3.3 javadoc is in Chinese [pulsar]
dlg99 opened a new issue, #22887: URL: https://github.com/apache/pulsar/issues/22887 ### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Read release policy - [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker. ### Version 3.3 ### Minimal reproduce step open https://pulsar.apache.org/api/client/3.3.x/org/apache/pulsar/client/api/MessageRouter as linked at https://pulsar.apache.org/docs/3.3.x/concepts-messaging/#routing-modes ### What did you expect to see? English docs ### What did you see instead? Chinese docs ### Anything else? _No response_ ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar-site) branch main updated: Rebuild: BUILD_ALL_VERSION=1
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/pulsar-site.git The following commit(s) were added to refs/heads/main by this push: new 80bc489f6df3 Rebuild: BUILD_ALL_VERSION=1 80bc489f6df3 is described below commit 80bc489f6df3b449f7284dc7fde28e810f920301 Author: Lari Hotari AuthorDate: Mon Jun 10 21:12:15 2024 +0300 Rebuild: BUILD_ALL_VERSION=1
[PR] [improve] Upgrade IPAddress to 5.5.0 [pulsar]
merlimat opened a new pull request, #22886: URL: https://github.com/apache/pulsar/pull/22886 ### Motivation Upgrading to latest version of IP address parsing lib to get fixes & improvements. ### Modifications ### Verifying this change - [ ] Make sure that the change passes the CI checks. *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Does this pull request potentially affect one of the following parts: *If the box was checked, please highlight the changes* - [ ] Dependencies (add or upgrade a dependency) - [ ] The public API - [ ] The schema - [ ] The default values of configurations - [ ] The threading model - [ ] The binary protocol - [ ] The REST endpoints - [ ] The admin CLI options - [ ] The metrics - [ ] Anything that affects deployment ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` ### Matching PR in forked repository PR in forked repository: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar-site) branch main updated: Improve clarity of the documentation banner
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/pulsar-site.git
The following commit(s) were added to refs/heads/main by this push:
new 868afe1f4f75 Improve clarity of the documentation banner
868afe1f4f75 is described below
commit 868afe1f4f75242cca397315e30e7738482ab9c4
Author: Lari Hotari
AuthorDate: Mon Jun 10 19:52:25 2024 +0300
Improve clarity of the documentation banner
---
src/theme/DocVersionBanner/index.js | 15 ---
1 file changed, 8 insertions(+), 7 deletions(-)
diff --git a/src/theme/DocVersionBanner/index.js
b/src/theme/DocVersionBanner/index.js
index f8c77655df3c..c61e8a9a70a7 100644
--- a/src/theme/DocVersionBanner/index.js
+++ b/src/theme/DocVersionBanner/index.js
@@ -15,7 +15,8 @@ import {
import BrowserOnly from "@docusaurus/BrowserOnly";
let versions = require("../../../versions.json");
const _latestVersion = versions[0];
-const _maintainedVersions = [versions[1], versions[2]];
+const _ltsVersion = '3.0.x';
+const _maintainedVersions = ['3.2.x', '3.0.x'];
function UnreleasedVersionLabel({ siteTitle, versionMetadata }) {
return (
{versionMetadata.label},
}}
>
- {"This is unreleased documentation for Next."}
+ {"This is documentation for the next unreleased version of Pulsar."}
);
}
@@ -40,7 +41,7 @@ function MaintainedVersionLabel({ siteTitle, versionMetadata
}) {
versionLabel: {versionMetadata.label},
}}
>
- {"This is the documentation for {versionLabel}."}
+ {"This is documentation for Pulsar {versionLabel}."}
);
}
@@ -55,7 +56,7 @@ function UnmaintainedVersionLabel({ siteTitle,
versionMetadata }) {
}}
>
{
-"This is the documentation for {versionLabel}, which is no longer
actively maintained."
+"This is documentation for Pulsar {versionLabel}, which is no longer
actively maintained."
}
);
@@ -101,7 +102,7 @@ function LatestVersionSuggestionLabel({ versionLabel, to,
onClick }) {
),
}}
>
- {"We recommend you use the {latestVersionLink}."}
+ {"We recommend that you use the {latestVersionLink} documentation."}
);
}
@@ -138,8 +139,7 @@ function DocVersionBannerEnabled({ className,
versionMetadata }) {
{
savePreferredVersionName(latestVersionSuggestion.name);
window.location.href = path;
@@ -155,6 +155,7 @@ export default function DocVersionBanner({ className }) {
{() => {
return versionMetadata.version != _latestVersion &&
+ versionMetadata.version != _ltsVersion &&
location.pathname.startsWith("/docs") ? (
(pulsar) branch master updated: [fix][cli] Fix Pulsar standalone "--wipe-data" (#22885)
This is an automated email from the ASF dual-hosted git repository.
mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new f6eceedbded [fix][cli] Fix Pulsar standalone "--wipe-data" (#22885)
f6eceedbded is described below
commit f6eceedbded53cded4dd751206ebb51d2867e978
Author: Lari Hotari
AuthorDate: Mon Jun 10 19:30:24 2024 +0300
[fix][cli] Fix Pulsar standalone "--wipe-data" (#22885)
---
.../src/main/java/org/apache/pulsar/PulsarStandalone.java| 9 -
.../org/apache/pulsar/zookeeper/LocalBookkeeperEnsemble.java | 2 ++
.../java/org/apache/pulsar/metadata/bookkeeper/BKCluster.java| 1 +
3 files changed, 11 insertions(+), 1 deletion(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
index 7f80aa29f53..d0118b06e7c 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
@@ -18,12 +18,14 @@
*/
package org.apache.pulsar;
+import static org.apache.commons.io.FileUtils.cleanDirectory;
import static org.apache.pulsar.common.naming.NamespaceName.SYSTEM_NAMESPACE;
import static
org.apache.pulsar.common.naming.SystemTopicNames.TRANSACTION_COORDINATOR_ASSIGN;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.collect.Sets;
import io.netty.util.internal.PlatformDependent;
import java.io.File;
+import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.List;
@@ -446,7 +448,12 @@ public class PulsarStandalone implements AutoCloseable {
void startBookieWithMetadataStore() throws Exception {
if (StringUtils.isBlank(metadataStoreUrl)){
log.info("Starting BK with RocksDb metadata store");
-metadataStoreUrl = "rocksdb://" +
Paths.get(metadataDir).toAbsolutePath();
+Path metadataDirPath = Paths.get(metadataDir);
+metadataStoreUrl = "rocksdb://" + metadataDirPath.toAbsolutePath();
+if (wipeData && Files.exists(metadataDirPath)) {
+log.info("Wiping RocksDb metadata store at {}",
metadataStoreUrl);
+cleanDirectory(metadataDirPath.toFile());
+}
} else {
log.info("Starting BK with metadata store: {}", metadataStoreUrl);
}
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/zookeeper/LocalBookkeeperEnsemble.java
b/pulsar-broker/src/main/java/org/apache/pulsar/zookeeper/LocalBookkeeperEnsemble.java
index e8a503c46e0..cf1a30951eb 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/zookeeper/LocalBookkeeperEnsemble.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/zookeeper/LocalBookkeeperEnsemble.java
@@ -194,6 +194,7 @@ public class LocalBookkeeperEnsemble {
: createTempDirectory("zktest");
if (this.clearOldData) {
+LOG.info("Wiping Zookeeper data directory at {}",
zkDataDir.getAbsolutePath());
cleanDirectory(zkDataDir);
}
@@ -291,6 +292,7 @@ public class LocalBookkeeperEnsemble {
: createTempDirectory("bk" + i + "test");
if (this.clearOldData) {
+LOG.info("Wiping Bookie data directory at {}",
bkDataDir.getAbsolutePath());
cleanDirectory(bkDataDir);
}
diff --git
a/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/BKCluster.java
b/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/BKCluster.java
index 8d3a90239ef..fe2b981ffe9 100644
---
a/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/BKCluster.java
+++
b/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/BKCluster.java
@@ -232,6 +232,7 @@ public class BKCluster implements AutoCloseable {
}
if (clusterConf.clearOldData && dataDir.exists()) {
+log.info("Wiping Bookie data directory at {}",
dataDir.getAbsolutePath());
cleanDirectory(dataDir);
}
Re: [I] Pulsar Standalone: --wipe-data does not work with RocksDB backend in 3.2.3 [pulsar]
merlimat closed issue #22881: Pulsar Standalone: --wipe-data does not work with RocksDB backend in 3.2.3 URL: https://github.com/apache/pulsar/issues/22881 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][cli] Fix Pulsar standalone "--wipe-data" [pulsar]
merlimat merged PR #22885: URL: https://github.com/apache/pulsar/pull/22885 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] Message loss on bookies and brokers restart [pulsar]
lhotari commented on issue #22709: URL: https://github.com/apache/pulsar/issues/22709#issuecomment-2158601494 > hey @lhotari, we'll contribute in that app to help to reproduce it and we'll let you know when we have something to share @PatrykWitkowski thanks, that will be helpful -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Upstream merge [pulsar-helm-chart]
hazbo closed pull request #505: Upstream merge URL: https://github.com/apache/pulsar-helm-chart/pull/505 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [improve][misc] PIP-351: Add options to Pulsar-Test client to support KeyStore based TLS [pulsar]
lhotari commented on PR #22692: URL: https://github.com/apache/pulsar/pull/22692#issuecomment-2158152104 PIP has been accepted and merged. https://lists.apache.org/thread/61l7bdhkjy0vy61934shmh2cphb6pp06 Good work @shasank112001 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch master updated: [improve][pip] PIP-351: Additional options for Pulsar-Test client to support KeyStore based TLS (#22694)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 540134c5b48 [improve][pip] PIP-351: Additional options for Pulsar-Test
client to support KeyStore based TLS (#22694)
540134c5b48 is described below
commit 540134c5b48ade391ee2a67235acb3416be1003c
Author: Shasank Sekhar Pandey <57795242+shasank112...@users.noreply.github.com>
AuthorDate: Mon Jun 10 13:51:51 2024 +0200
[improve][pip] PIP-351: Additional options for Pulsar-Test client to
support KeyStore based TLS (#22694)
---
pip/pip-351.md | 166 +
1 file changed, 166 insertions(+)
diff --git a/pip/pip-351.md b/pip/pip-351.md
new file mode 100644
index 000..17f88b48955
--- /dev/null
+++ b/pip/pip-351.md
@@ -0,0 +1,166 @@
+
+
+# PIP-351: Additional options for Pulsar-Test client to support KeyStore based
TLS
+
+# Background knowledge
+
+
+
+In both Pulsar Client and Pulsar Admin, we support the use of KeyStores. This
feature is provided by means of the boolean
+"useKeyStoreTls". The boolean is also the only way authentication mechanisms
such as AuthenticationKeyStoreTls can be utilised
+properly, as the logic to use keystores for SSL Connections, from either
ClientConfigurationData stored in Pulsar Admin/Client
+or AuthData hinges on the "useKeyStoreTls" boolean as can be seen below:
+
+AsyncHttpConnector.java
+```java
+if (conf.isUseKeyStoreTls()) {
+KeyStoreParams params = authData.hasDataForTls() ?
authData.getTlsKeyStoreParams() :
+new KeyStoreParams(conf.getTlsKeyStoreType(),
conf.getTlsKeyStorePath(),
+conf.getTlsKeyStorePassword());
+
+final SSLContext sslCtx = KeyStoreSSLContext.createClientSslContext(
+conf.getSslProvider(),
+params.getKeyStoreType(),
+params.getKeyStorePath(),
+params.getKeyStorePassword(),
+conf.isTlsAllowInsecureConnection(),
+conf.getTlsTrustStoreType(),
+conf.getTlsTrustStorePath(),
+conf.getTlsTrustStorePassword(),
+conf.getTlsCiphers(),
+conf.getTlsProtocols());
+
+JsseSslEngineFactory sslEngineFactory = new JsseSslEngineFactory(sslCtx);
+confBuilder.setSslEngineFactory(sslEngineFactory);
+}
+```
+
+None of these options can be currently configured when using Pulsar Test
client.
+
+# Motivation
+
+
+
+As we already let users both extend authentication and use just the keystore
and truststore properties to set up mTLS
+connections, without using any authentication plugin class, a lot of them
might want to use this method of authentication
+during Performance Testing as well.
+
+I understand that currently mTLS (for testing purposes) can be achieved by
using trust and client certificates.
+However, the issue of users extending authentication plugin classes and
utilizing keystores is still not covered
+with the current options. Therefore, I propose we make these already existing
options be configured in test clients,
+increasing its usability.
+
+# Goals
+
+## In Scope
+
+Create new Arguments for the following properties, in
PerformanceBaseArguments.java :
+1. useKeyStoreTls
+2. trustStoreType
+3. trustStorePath
+4. trustStorePass
+5. keyStoreType
+6. keyStorePath
+7. keyStorePass
+
+Update the code to change between TrustCerts and TrustStore based on
useKeyStoreTls.
+
+
+
+[//]: # (## Out of Scope)
+
+
+
+
+[//]: # (# High Level Design)
+
+
+
+# Detailed Design
+
+## Design & Implementation Details
+
+
+
+Add the options for utilizing keystores as part of performance base arguments,
along with forwarding their values
+to the client/admin builders.
+
+## Public-facing Changes
+
+
+
+### CLI
+
+All places we utilize Pulsar Test client, for example Pulsar-Perf will have
the following new options:
+
+1. --use-keystore-tls Default value = false
+2. --truststore-type Default value = JKS, Possible values = JKS, PKCS12
+3. --truststore-path Default value = ""
+4. --truststore-pass Default value = ""
+5. --keystore-type Default value = JKS, Possible values = JKS, PKCS12
+6. --keystore-path Default value = ""
+7. --keystore-pass Default value = ""
+
+
+
+# Backward & Forward Compatibility
+
+The change will not affect any previous releases. The options can also be
brought to previous versions, however, I have
+noticed that Pulsar has moved away from JCommander in Version 3.2.x to Picocli
(currently in master)
+Therefore, to add these options to previous versions, the code has to be
replicated to those versions.
Re: [PR] [improve][pip] PIP-351: Additional options for Pulsar-Test client to support KeyStore based TLS [pulsar]
lhotari merged PR #22694: URL: https://github.com/apache/pulsar/pull/22694 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] Message loss on bookies and brokers restart [pulsar]
PatrykWitkowski commented on issue #22709: URL: https://github.com/apache/pulsar/issues/22709#issuecomment-2158083428 hey @lhotari, we'll contribute in that app to help to reproduce it and we'll let you know when we have something to share -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar-dotpulsar) branch master updated: Update CHANGELOG.md
This is an automated email from the ASF dual-hosted git repository. djensen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar-dotpulsar.git The following commit(s) were added to refs/heads/master by this push: new 3889bf0 Update CHANGELOG.md 3889bf0 is described below commit 3889bf098c92f3c57e71728036ab022a9b94fe0b Author: entvex <1580435+ent...@users.noreply.github.com> AuthorDate: Mon Jun 10 12:42:32 2024 +0200 Update CHANGELOG.md --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c23db24..0c0a1d8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [3.3.0-rc.1] - 2024-06-07 +## [3.3.0] - 2024-06-10 ### Added
(pulsar-dotpulsar) tag 3.3.0 created (now c95ee48)
This is an automated email from the ASF dual-hosted git repository. djensen pushed a change to tag 3.3.0 in repository https://gitbox.apache.org/repos/asf/pulsar-dotpulsar.git at c95ee48 (commit) No new revisions were added by this update.
svn commit: r69616 - /dev/pulsar/pulsar-dotpulsar-3.3.0-rc.1/ /release/pulsar/pulsar-dotpulsar-3.3.0/
Author: djensen Date: Mon Jun 10 10:34:56 2024 New Revision: 69616 Log: release 3.3.0 Added: release/pulsar/pulsar-dotpulsar-3.3.0/ - copied from r69615, dev/pulsar/pulsar-dotpulsar-3.3.0-rc.1/ Removed: dev/pulsar/pulsar-dotpulsar-3.3.0-rc.1/
Re: [I] [Bug] one topic suddenly cannot be consumed,others is ok [pulsar]
danielnesaraj commented on issue #21082: URL: https://github.com/apache/pulsar/issues/21082#issuecomment-2157473672 I think the reverse is true; we scale the consumers based on the backlog size. So as the backlog started increasing, there would have been more consumers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [improve][broker] Optimize PersistentTopic.getLastDispatchablePosition [pulsar]
codecov-commenter commented on PR #22707: URL: https://github.com/apache/pulsar/pull/22707#issuecomment-2157397197 ## [Codecov](https://app.codecov.io/gh/apache/pulsar/pull/22707?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report Attention: Patch coverage is `73.17073%` with `11 lines` in your changes missing coverage. Please review. > Project coverage is 73.29%. Comparing base [(`bbc6224`)](https://app.codecov.io/gh/apache/pulsar/commit/bbc62245c5ddba1de4b1e7cee4ab49334bc36277?dropdown=coverage=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) to head [(`78b`)](https://app.codecov.io/gh/apache/pulsar/commit/78b99f0c26bee8604ba2517554003688db81?dropdown=coverage=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache). > Report is 366 commits behind head on master. Additional details and impacted files [](https://app.codecov.io/gh/apache/pulsar/pull/22707?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) ```diff @@ Coverage Diff @@ ## master #22707 +/- ## - Coverage 73.57% 73.29% -0.29% - Complexity3262433015 +391 Files 1877 1891 +14 Lines139502 141969+2467 Branches 1529915571 +272 + Hits 102638 104051+1413 - Misses2890829893 +985 - Partials 7956 8025 +69 ``` | [Flag](https://app.codecov.io/gh/apache/pulsar/pull/22707/flags?src=pr=flags_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Coverage Δ | | |---|---|---| | [inttests](https://app.codecov.io/gh/apache/pulsar/pull/22707/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `27.21% <56.09%> (+2.62%)` | :arrow_up: | | [systests](https://app.codecov.io/gh/apache/pulsar/pull/22707/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `24.70% <39.02%> (+0.37%)` | :arrow_up: | | [unittests](https://app.codecov.io/gh/apache/pulsar/pull/22707/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `72.30% <73.17%> (-0.55%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files](https://app.codecov.io/gh/apache/pulsar/pull/22707?dropdown=coverage=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Coverage Δ | | |---|---|---| | [...ransaction/buffer/impl/TopicTransactionBuffer.java](https://app.codecov.io/gh/apache/pulsar/pull/22707?src=pr=tree=pulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Ftransaction%2Fbuffer%2Fimpl%2FTopicTransactionBuffer.java_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci90cmFuc2FjdGlvbi9idWZmZXIvaW1wbC9Ub3BpY1RyYW5zYWN0aW9uQnVmZmVyLmphdmE=) | `88.21% <100.00%> (+0.46%)` | :arrow_up: | | [...sar/broker/service/persistent/PersistentTopic.java](https://app.codecov.io/gh/apache/pulsar/pull/22707?src=pr=tree=pulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fservice%2Fpersistent%2FPersistentTopic.java_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zZXJ2aWNlL3BlcnNpc3RlbnQvUGVyc2lzdGVudFRvcGljLmphdmE=) | `79.12% <90.90%> (+0.66%)` | :arrow_up: | | [...nsaction/buffer/impl/TransactionBufferDisable.java](https://app.codecov.io/gh/apache/pulsar/pull/22707?src=pr=tree=pulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Ftransaction%2Fbuffer%2Fimpl%2FTransactionBufferDisable.java_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci90cmFuc2FjdGlvbi9idWZmZXIvaW1wbC9UcmFuc2FjdGlvbkJ1ZmZlckRpc2FibGUuamF2YQ==) | `57.57% <71.42%> (+1.05%)` | :arrow_up: | |
