[GitHub] [pulsar] codelipenghui commented on pull request #7042: [Issue 6957][pulsar-io-debezium] Support CDC Connector for SqlServer
codelipenghui commented on pull request #7042: URL: https://github.com/apache/pulsar/pull/7042#issuecomment-634440868 @huangdx0726 Would you please help take a look at the failed CI tests? Looks related to this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui merged pull request #6990: Update the doc for deleting inactive topic when subscriptions caught up
codelipenghui merged pull request #6990: URL: https://github.com/apache/pulsar/pull/6990 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: Update the broker configuration file in the Reference > Pulsar configuration for the issue apache/pulsar#6087 (#6990)
This is an automated email from the ASF dual-hosted git repository. penghui pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new 2423c9f Update the broker configuration file in the Reference > Pulsar configuration for the issue apache/pulsar#6087 (#6990) 2423c9f is described below commit 2423c9f307f26302503da0320fcaec754547bc5e Author: HuanliMeng <48120384+huanli-m...@users.noreply.github.com> AuthorDate: Wed May 27 13:40:14 2020 +0800 Update the broker configuration file in the Reference > Pulsar configuration for the issue apache/pulsar#6087 (#6990) --- site2/docs/reference-configuration.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/site2/docs/reference-configuration.md b/site2/docs/reference-configuration.md index b1e893f..893ea0c 100644 --- a/site2/docs/reference-configuration.md +++ b/site2/docs/reference-configuration.md @@ -136,6 +136,8 @@ Pulsar brokers are responsible for handling incoming messages from producers, di |defaultNumPartitions| The number of partitioned topics that is allowed to be automatically created if `allowAutoTopicCreationType` is partitioned |1| |brokerDeleteInactiveTopicsEnabled| Enable the deletion of inactive topics |true| |brokerDeleteInactiveTopicsFrequencySeconds| How often to check for inactive topics |60| +| brokerDeleteInactiveTopicsMode | Set the mode to delete inactive topics. `delete_when_no_subscriptions`: delete the topic which has no subscriptions or active producers. `delete_when_subscriptions_caught_up`: delete the topic whose subscriptions have no backlogs and which has no active producers or consumers. | `delete_when_no_subscriptions` | +| brokerDeleteInactiveTopicsMaxInactiveDurationSeconds | Set the maximum duration for inactive topics. If it is not specified, the `brokerDeleteInactiveTopicsFrequencySeconds` parameter is adopted. | N/A | |messageExpiryCheckIntervalInMinutes| How frequently to proactively check and purge expired messages |5| |brokerServiceCompactionMonitorIntervalInSeconds| Interval between checks to see if topics with compaction policies need to be compacted |60| |activeConsumerFailoverDelayTimeMillis| How long to delay rewinding cursor and dispatching messages when active consumer is changed. |1000|
[GitHub] [pulsar] codelipenghui merged pull request #6856: fix add partition failed when bundle unloaded
codelipenghui merged pull request #6856: URL: https://github.com/apache/pulsar/pull/6856 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: fix add partition failed when bundle unloaded (#6856)
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new bbec7eb fix add partition failed when bundle unloaded (#6856)
bbec7eb is described below
commit bbec7eb6ca30bef10b83d46839ef152e3267c656
Author: hangc0276
AuthorDate: Wed May 27 13:35:45 2020 +0800
fix add partition failed when bundle unloaded (#6856)
### Motivation
When a topic with high input/output load, we will get add partition failed,
the failed log as follows:
```
Failed to perform http post request:
org.asynchttpclient.handler.MaxRedirectException: Maximum redirect reached: 5
null
Reason: org.asynchttpclient.handler.MaxRedirectException: Maximum redirect
reached: 5
```
### Bug description
The reason is when the topic with high load, the topic's bundle will be
unload. In the same time, we call pulsar admin to add partition for the topic,
the request will post to one broker A, broker A can't find the topic-bundle's
owner, it will redirect the request to the leader broker B, broker B find a
candidate broker C to own the bundle, and redirect request to broker C with
authoritative flag.
However, broker C can't receive the authoritative flag (that's the bug),
and can't find the topic-bundle's owner and has no authoritative flag, so it
redirect the request to the leader broker B and goes on in cycle. In the end it
reaches the max redirect limit and failed.
---
.../org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java | 4 ++--
.../java/org/apache/pulsar/broker/admin/v1/PersistentTopics.java | 8 +---
.../java/org/apache/pulsar/broker/admin/v2/PersistentTopics.java | 4 +++-
.../java/org/apache/pulsar/broker/admin/PersistentTopicsTest.java | 2 +-
4 files changed, 11 insertions(+), 7 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java
index ece5371..a052191 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java
@@ -468,8 +468,8 @@ public class PersistentTopicsBase extends AdminResource {
*
* @param numPartitions
*/
-protected void internalUpdatePartitionedTopic(int numPartitions, boolean
updateLocalTopicOnly) {
-validateWriteOperationOnTopic(false);
+protected void internalUpdatePartitionedTopic(int numPartitions, boolean
updateLocalTopicOnly, boolean authoritative) {
+validateWriteOperationOnTopic(authoritative);
// Only do the validation if it's the first hop.
if (!updateLocalTopicOnly) {
validatePartitionTopicUpdate(topicName.getLocalName(),
numPartitions);
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v1/PersistentTopics.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v1/PersistentTopics.java
index 3c56f1c..c3003e5 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v1/PersistentTopics.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v1/PersistentTopics.java
@@ -46,8 +46,6 @@ import org.apache.pulsar.client.api.MessageId;
import org.apache.pulsar.client.impl.MessageIdImpl;
import org.apache.pulsar.common.partition.PartitionedTopicMetadata;
import org.apache.pulsar.common.policies.data.AuthAction;
-import org.apache.pulsar.common.policies.data.PartitionedTopicInternalStats;
-import org.apache.pulsar.common.policies.data.PartitionedTopicStats;
import org.apache.pulsar.common.policies.data.PersistentOfflineTopicStats;
import org.apache.pulsar.common.policies.data.PersistentTopicInternalStats;
import org.apache.pulsar.common.policies.data.TopicStats;
@@ -57,6 +55,8 @@ import io.swagger.annotations.Api;
import io.swagger.annotations.ApiOperation;
import io.swagger.annotations.ApiResponse;
import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.ApiParam;
+
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -186,9 +186,11 @@ public class PersistentTopics extends PersistentTopicsBase
{
public void updatePartitionedTopic(@PathParam("property") String property,
@PathParam("cluster") String cluster,
@PathParam("namespace") String namespace, @PathParam("topic")
@Encoded String encodedTopic,
@QueryParam("updateLocalTopicOnly") @DefaultValue("false") boolean
updateLocalTopicOnly,
+@ApiParam(value = "Is authentication required to perform this
operation")
+@QueryParam("authoritative") @DefaultValue("false") boolean
authoritative,
int numPartitions) {
validateTopicName(property, cluster,
[GitHub] [pulsar] codelipenghui commented on issue #6838: java.lang.OutOfMemoryError: GC overhead limit exceeded' is observed when creating producers on a non-existent topic
codelipenghui commented on issue #6838: URL: https://github.com/apache/pulsar/issues/6838#issuecomment-634438687 @devprashantksinha Did you upgrade the Pulsar broker? #6879 makes some changes on the broker side. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #6719: fix consumer fetch message number maps to read entry number bug and expose avgMessagesPerEntry metric
codelipenghui commented on pull request #6719: URL: https://github.com/apache/pulsar/pull/6719#issuecomment-634437448 /pulsarbot run-failure-checks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui merged pull request #7045: Add msg and bytes counter stats to broker-stats/topics
codelipenghui merged pull request #7045: URL: https://github.com/apache/pulsar/pull/7045 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui merged pull request #7044: Add test time for perf consumer and perf reader
codelipenghui merged pull request #7044: URL: https://github.com/apache/pulsar/pull/7044 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] rocketraman opened a new pull request #7040: [docs] minor formatting fix
rocketraman opened a new pull request #7040: URL: https://github.com/apache/pulsar/pull/7040 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on pull request #6827: [Issue 6822][C++] Fix segmentation fault if handleSendTimeout() is called after closeAsync()
BewareMyPower commented on pull request #6827: URL: https://github.com/apache/pulsar/pull/6827#issuecomment-633589986 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] sijie merged pull request #7037: [docs] Specify namespace when launching pulsar-manager
sijie merged pull request #7037: URL: https://github.com/apache/pulsar/pull/7037 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] murong00 commented on pull request #6998: Add multi-hosts example for bookkeeperMetadataServiceUri
murong00 commented on pull request #6998: URL: https://github.com/apache/pulsar/pull/6998#issuecomment-633785601 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] zplinuxlover commented on a change in pull request #6903: PIP-61: Advertise multiple addresses
zplinuxlover commented on a change in pull request #6903:
URL: https://github.com/apache/pulsar/pull/6903#discussion_r429967851
##
File path:
pulsar-broker/src/main/java/org/apache/pulsar/broker/lookup/TopicLookupBase.java
##
@@ -156,6 +156,31 @@ protected String internalGetNamespaceBundle(TopicName
topicName) {
*/
public static CompletableFuture lookupTopicAsync(PulsarService
pulsarService, TopicName topicName,
boolean authoritative, String clientAppId,
AuthenticationDataSource authenticationData, long requestId) {
+return lookupTopicAsync(pulsarService, topicName, authoritative,
clientAppId, authenticationData, requestId, null);
+}
+
+/**
+ *
+ * Lookup broker-service address for a given namespace-bundle which
contains given topic.
+ *
+ * a. Returns broker-address if namespace-bundle is already owned by any
broker b. If current-broker receives
+ * lookup-request and if it's not a leader then current broker redirects
request to leader by returning
+ * leader-service address. c. If current-broker is leader then it finds
out least-loaded broker to own namespace
+ * bundle and redirects request by returning least-loaded broker. d. If
current-broker receives request to own the
+ * namespace-bundle then it owns a bundle and returns success(connect)
response to client.
Review comment:
I have format the comment
##
File path:
pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
##
@@ -36,11 +36,7 @@
import java.lang.reflect.Method;
import java.net.InetSocketAddress;
import java.net.URI;
-import java.util.Collections;
-import java.util.List;
-import java.util.Map;
-import java.util.Objects;
-import java.util.Optional;
+import java.util.*;
Review comment:
resolve the problem
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] sijie commented on pull request #6566: [PIP-60] [Proxy-Server] Support SNI routing to support various proxy-server in pulsar
sijie commented on pull request #6566: URL: https://github.com/apache/pulsar/pull/6566#issuecomment-633846180 /pulsarbot run-failure-checks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] k2la opened a new issue #7041: Producers failed to open when leader broker shut down
k2la opened a new issue #7041:
URL: https://github.com/apache/pulsar/issues/7041
When a leader broker shut down, producers that connected with the broker
failed to open on a new broker.
According to the leader broker log, the broker unloaded bundles and closed
producers.
Also, according to logs of other brokers, another broker became leader
broker and bundles were loaded.
However, producers that connected with the old leader broker reconnected
with a new broker but some producers of ones failed to open on the new broker.
After the producers reconnected with the new broker, they didn't send
`CommandProducer` messages and stop.
https://github.com/apache/pulsar/blob/d55bc00f34a2fa763a3756fa0adbb1366ae319bd/pulsar-common/src/main/proto/PulsarApi.proto#L406-L430
Expected behavior
When producers reconnect with a new broker, open on the broker.
Actual behavior
Some producers failed to open on the new broker.
Steps to reproduce
We tried but haven't reproduced yet.
System configuration
OS(Broker): CentOS 7.7
Pulsar Broker: 2.3.2
Pulsar Client Java: 2.3.2
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] 315157973 commented on issue #7032: Pulsar supporting DLQ for sources/sinks
315157973 commented on issue #7032: URL: https://github.com/apache/pulsar/issues/7032#issuecomment-634382545 Please assign it to me This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on pull request #6796: [C++] Support cumulative acknowledgement when consuming partitioned topics
BewareMyPower commented on pull request #6796: URL: https://github.com/apache/pulsar/pull/6796#issuecomment-633590154 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #6980: Update the function CLI description
codelipenghui commented on pull request #6980: URL: https://github.com/apache/pulsar/pull/6980#issuecomment-633600537 @Huanli-Meng Would you please rebase the master branch since there are some new changes related to the failed CI tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] sijie commented on pull request #6992: [Issue 6921][pulsar-broker-common] Replaced "Paths.get(...).getParent…
sijie commented on pull request #6992: URL: https://github.com/apache/pulsar/pull/6992#issuecomment-633844004 /puslarbot run-failure-checks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] sijie edited a comment on issue #3727: Support for Distributed Tracing with OpenTracing
sijie edited a comment on issue #3727: URL: https://github.com/apache/pulsar/issues/3727#issuecomment-634282445 For people who are looking for open tracing integration, we have open-sourced our open-tracking interceptor for Pulsar client. documentation: https://hub.streamnative.io/monitoring/opentracing-pulsar-client/0.1.0 source code: https://github.com/streamnative/pulsar-tracing This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] BewareMyPower removed a comment on pull request #6827: [Issue 6822][C++] Fix segmentation fault if handleSendTimeout() is called after closeAsync()
BewareMyPower removed a comment on pull request #6827: URL: https://github.com/apache/pulsar/pull/6827#issuecomment-631356794 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] curtiscook opened a new issue #7050: [python] pulsar-client@2.5.2's dep on enum34@1.1.9 may break build systems
curtiscook opened a new issue #7050: URL: https://github.com/apache/pulsar/issues/7050 Enum34 appears to have had a regression of the below issue in 1.1.9 https://bitbucket.org/stoneleaf/enum34/issues/31/make-the-install-a-noop-on-python36 This breaks builds due to `AttributeError: module 'enum' has no attribute 'IntFlag'` Related poetry issue: https://github.com/python-poetry/poetry/issues/1122#issuecomment-628037127 Suggest using enum34@1.1.8 until this is fixed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #4955: [PIP-39] Introduce system topic and topic policies service
codelipenghui commented on pull request #4955: URL: https://github.com/apache/pulsar/pull/4955#issuecomment-634037465 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Omega-Ariston opened a new pull request #7038: [website][issue7002]fixes RESTAPIs and Cli button not working.
Omega-Ariston opened a new pull request #7038:
URL: https://github.com/apache/pulsar/pull/7038
Fixes #7002
### Motivation
the `REST APIs` and `Cli` button only work in English ver. website.
### Modifications
Currently, the site retrieve user selected version by
`document.querySelector('a[href="/en/versions"]')`. Therefore, when the
language changes, this path will no longer exists(it will become
`a[href="/zh-CN/versions"]` for Chinese ver. for example).
So I changed another way that achieves the same goal, with the help of
`logo` class.
The downside is that this new line of code might be ambiguous. So I added
another line to comment it.
### Verifying this change
- [ yes ] Make sure that the change passes the CI checks.
*(Please pick either of the following options)*
This change is a trivial rework / code cleanup without any test coverage.
### Does this pull request potentially affect one of the following parts:
*If `yes` was chosen, please highlight the changes*
- Dependencies (does it add or upgrade a dependency): (no)
- The public API: (no)
- The schema: (no)
- The default values of configurations: (no)
- The wire protocol: (no)
- The rest endpoints: (no)
- The admin cli options: (no)
- Anything that affects deployment: (no)
### Documentation
- Does this pull request introduce a new feature? (no)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #4400: PIP 37: [pulsar-client] support large message size
codelipenghui commented on pull request #4400: URL: https://github.com/apache/pulsar/pull/4400#issuecomment-633608821 @rdhabalia Would you please resolve the conflicts? So that we can onboard this PR in 2.6.0 release. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #3497: Upgrade guava to 27.0.1
codelipenghui commented on pull request #3497: URL: https://github.com/apache/pulsar/pull/3497#issuecomment-633607021 move to 2.7.0 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] charvakcpatel007 closed pull request #7026: Billing metric update to zookeeper store
charvakcpatel007 closed pull request #7026: URL: https://github.com/apache/pulsar/pull/7026 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] rocketraman opened a new pull request #7039: [docs] Fix spelling error
rocketraman opened a new pull request #7039: URL: https://github.com/apache/pulsar/pull/7039 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #7045: Add msg and bytes counter stats to broker-stats/topics
codelipenghui commented on pull request #7045: URL: https://github.com/apache/pulsar/pull/7045#issuecomment-634104572 /pulsarbot run-failure-checks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] rivernate commented on pull request #6911: Split KafkaConnectSource into an abstract and impl
rivernate commented on pull request #6911: URL: https://github.com/apache/pulsar/pull/6911#issuecomment-634251318 /pulsarbot run-failure-checks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] sijie commented on pull request #6799: optional auth method name header in http authentication
sijie commented on pull request #6799: URL: https://github.com/apache/pulsar/pull/6799#issuecomment-633845920 /pulsarbot run-failure-checks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] sijie merged pull request #7035: [docs] Update admin-api-brokers.md
sijie merged pull request #7035: URL: https://github.com/apache/pulsar/pull/7035 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] sijie commented on pull request #7014: [C++] Expose zstd and Snappy compression to C API
sijie commented on pull request #7014: URL: https://github.com/apache/pulsar/pull/7014#issuecomment-633837768 /pulsarbot run-failure-checks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #7043: Pulsar Admin fails on creating a new topic if you have multiple hosts in a URL
codelipenghui commented on issue #7043: URL: https://github.com/apache/pulsar/issues/7043#issuecomment-634035433 I can't reproduce it on 2.5.2 Java Client and master branch. Are there any other special configurations? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lukestephenson closed issue #6950: loadBalancerSourceRanges on helm templates for pulsar manager and grafana
lukestephenson closed issue #6950: URL: https://github.com/apache/pulsar/issues/6950 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] d-eggert opened a new issue #7049: Make WebSocketService.MaxTextFrameSize configurable
d-eggert opened a new issue #7049: URL: https://github.com/apache/pulsar/issues/7049 **Is your feature request related to a problem? Please describe.** We use the websocket interface to pulsar. Our current use case demands the transmission of big messages (10MB or even more), which is not possible due to the hardcoded limit in `WebSocketService` **Describe the solution you'd like** Make this value configurable, e.g. via websocket.conf **Describe alternatives you've considered** Maybe increase the hardcoded default. **Additional context** None. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] hangc0276 commented on pull request #6856: fix add partition failed when bundle unloaded
hangc0276 commented on pull request #6856: URL: https://github.com/apache/pulsar/pull/6856#issuecomment-633973470 /pulsarbot run-failure-checks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] sijie merged pull request #7040: [docs] minor formatting fix
sijie merged pull request #7040: URL: https://github.com/apache/pulsar/pull/7040 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] vzhikserg commented on issue #6587: Requesting Pulsar to support IoT protocols - STOMP, AMQP, MQTT, WSS
vzhikserg commented on issue #6587: URL: https://github.com/apache/pulsar/issues/6587#issuecomment-633604083 This issue contains too many requests in one. I would suggest to create 4 separate issues (one for each protocol) and close this one. The issues can be closed as soon as the handlers are implemented and we don't have to wait till all other handlers are finished. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] sijie commented on pull request #7027: [pulsar-client] Fix typo in exception message
sijie commented on pull request #7027: URL: https://github.com/apache/pulsar/pull/7027#issuecomment-633836276 /pulsarbot run-failure-checks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] sijie commented on pull request #7026: Billing metric update to zookeeper store
sijie commented on pull request #7026: URL: https://github.com/apache/pulsar/pull/7026#issuecomment-633836069 What is the motivation of this change? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #2602: ManagedLedger should share BK worker pool threads
codelipenghui commented on pull request #2602: URL: https://github.com/apache/pulsar/pull/2602#issuecomment-633606530 move to 2.7.0 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #7032: Pulsar supporting DLQ for sources/sinks
codelipenghui commented on issue #7032: URL: https://github.com/apache/pulsar/issues/7032#issuecomment-633619648 @315157973 is working on this issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] abhilashmandaliya opened a new issue #7043: Pulsar Admin fails on creating a new topic if you have multiple hosts in a URL
abhilashmandaliya opened a new issue #7043:
URL: https://github.com/apache/pulsar/issues/7043
**Describe the bug**
When you have multiple pulsar brokers and try to create a topic via Java
client using multi-host URL, it fails. My admin URL is as follow:
`http://192.168.254.1:34099,192.168.254.1:34102`
**To Reproduce**
Steps to reproduce the behavior:
1. Run a pulsar cluster with at least 2 nodes
2. Use a java client with multi-host admin URL to connect
3. Invoke this code:
`pulsarAdmin.topics().createPartitionedTopic("persistent://public/default/myTopic",
2);`
You will get the following exception:
`java.lang.NumberFormatException: For input string:
"34065,192.168.254.1:34063"
org.apache.pulsar.client.admin.PulsarAdminException:
java.lang.NumberFormatException: For input string: "34065,192.168.254.1:34063"
at
org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:228)
at
org.apache.pulsar.client.admin.internal.BaseResource$1.failed(BaseResource.java:130)
at
org.apache.pulsar.shade.org.glassfish.jersey.client.JerseyInvocation$4.failed(JerseyInvocation.java:1030)
at
org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime.processFailure(ClientRuntime.java:231)
at
org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime.lambda$null$3(ClientRuntime.java:189)
at
org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
at
org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
at
org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors.process(Errors.java:316)
at
org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors.process(Errors.java:298)
at
org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors.process(Errors.java:268)
at
org.apache.pulsar.shade.org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:312)
at
org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime.lambda$createRunnableForAsyncProcessing$4(ClientRuntime.java:163)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NumberFormatException: For input string:
"34065,192.168.254.1:34063"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.parseInt(Integer.java:615)
at
org.apache.pulsar.shade.org.asynchttpclient.uri.UriParser.computeRegularHostPort(UriParser.java:216)
at
org.apache.pulsar.shade.org.asynchttpclient.uri.UriParser.parseAuthority(UriParser.java:299)
at
org.apache.pulsar.shade.org.asynchttpclient.uri.UriParser.parse(UriParser.java:358)
at
org.apache.pulsar.shade.org.asynchttpclient.uri.Uri.create(Uri.java:67)
at
org.apache.pulsar.shade.org.asynchttpclient.uri.Uri.create(Uri.java:62)
at
org.apache.pulsar.shade.org.asynchttpclient.RequestBuilderBase.setUrl(RequestBuilderBase.java:147)
at
org.apache.pulsar.shade.org.asynchttpclient.DefaultAsyncHttpClient.requestBuilder(DefaultAsyncHttpClient.java:294)
at
org.apache.pulsar.shade.org.asynchttpclient.DefaultAsyncHttpClient.prepare(DefaultAsyncHttpClient.java:133)
at
org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.apply(AsyncHttpConnector.java:240)
at
org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime.lambda$null$3(ClientRuntime.java:187)
... 12 more`
**Expected behaviour**
Should not throw an exception and create a topic
**System Information**
- Docker version: 2.3.0.2 (45183)
- Ubuntu image: Ubuntu 16.04.6 LTS (Xenial Xerus)
- Pulsar version: 2.5.0
- Pulsar Java Client version: 2.5.2
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] murong00 removed a comment on pull request #6998: Add multi-hosts example for bookkeeperMetadataServiceUri
murong00 removed a comment on pull request #6998: URL: https://github.com/apache/pulsar/pull/6998#issuecomment-633785601 /pulsarbot run-failure-checks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] ckdarby commented on issue #6969: Brokers crash if all bookies are full
ckdarby commented on issue #6969: URL: https://github.com/apache/pulsar/issues/6969#issuecomment-633729334 @trexinc Were you using "small volumes" with large ingestion? Better put, could you fill ~10% of your total bookies in < 10 seconds? We faced a similar issue where the cluster filled all the bookies before the ReadOnly safety check could even be performed and the cluster went into a state of being partially unusable for some functions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #6998: Add multi-hosts example for bookkeeperMetadataServiceUri
codelipenghui commented on pull request #6998: URL: https://github.com/apache/pulsar/pull/6998#issuecomment-633599898 @murong00 Would you please rebase the master branch since there are some new changes related to the failed CI tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #7044: Add test time for perf consumer and perf reader
codelipenghui commented on pull request #7044: URL: https://github.com/apache/pulsar/pull/7044#issuecomment-634024110 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] huangdx0726 opened a new pull request #7042: [Issue 6957][pulsar-io-debezium] Support CDC Connector for SqlServer
huangdx0726 opened a new pull request #7042: URL: https://github.com/apache/pulsar/pull/7042 Fixes #6957 ### Motivation Add debezium Sqlserver source ### Modifications Add debezium Sqlserver source ### Verifying this change - [ ] Make sure that the change passes the CI checks. *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Does this pull request potentially affect one of the following parts: *If `yes` was chosen, please highlight the changes* - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API: (yes / no) - The schema: (yes / no / don't know) - The default values of configurations: (yes / no) - The wire protocol: (yes / no) - The rest endpoints: (yes / no) - The admin cli options: (yes / no) - Anything that affects deployment: (yes / no / don't know) ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a followup issue for adding the documentation This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] ckdarby commented on issue #6950: loadBalancerSourceRanges on helm templates for pulsar manager and grafana
ckdarby commented on issue #6950: URL: https://github.com/apache/pulsar/issues/6950#issuecomment-633729627 @lukestephenson I believe Sijie has mentioned it'll happen in 2.6.0. I see your PR merged in https://github.com/apache/pulsar-helm-chart/commit/96dbab924fcad80c9586bf3f3c10445c1dec8ee3 Can you close this issue here :) ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on a change in pull request #7018: [Doc] Update for acknowledgment at batch index level
Anonymitaet commented on a change in pull request #7018: URL: https://github.com/apache/pulsar/pull/7018#discussion_r430308755 ## File path: site2/docs/concepts-messaging.md ## @@ -49,14 +49,15 @@ Messages published by producers can be compressed during transportation in order ### Batching -If batching is enabled, the producer will accumulate and send a batch of messages in a single request. Batch size is defined by the maximum number of messages and maximum publish latency. +When batching is enabled, the producer accumulates and sends a batch of messages in a single request. The batch size is defined by the maximum number of messages and the maximum publish latency. Therefore, the backlog size represents the the total number of batches instead of the total number of messages. Review comment: ```suggestion When batching is enabled, the producer accumulates and sends a batch of messages in a single request. The batch size is defined by the maximum number of messages and the maximum publish latency. Therefore, the backlog size represents the total number of batches instead of the total number of messages. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] srkukarni merged pull request #7039: [docs] Fix spelling error
srkukarni merged pull request #7039: URL: https://github.com/apache/pulsar/pull/7039 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] sijie merged pull request #7020: Make the json node field in GenericRecord public
sijie merged pull request #7020: URL: https://github.com/apache/pulsar/pull/7020 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-go] yarthur1 closed issue #259: problem about sendRequest fail to add batchbuilder
yarthur1 closed issue #259: URL: https://github.com/apache/pulsar-client-go/issues/259 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #3560: Remove broker-shaded artifacts
codelipenghui commented on pull request #3560: URL: https://github.com/apache/pulsar/pull/3560#issuecomment-633607184 move to 2.7.0 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #7050: [python] pulsar-client@2.5.2's dep on enum34@1.1.9 may break build systems
sijie commented on issue #7050: URL: https://github.com/apache/pulsar/issues/7050#issuecomment-634378604 @curtiscook since you have already found the issue, are you interested in submitting a pull request to address the issue? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] devprashantksinha commented on issue #6838: java.lang.OutOfMemoryError: GC overhead limit exceeded' is observed when creating producers on a non-existent topic
devprashantksinha commented on issue #6838: URL: https://github.com/apache/pulsar/issues/6838#issuecomment-634006722 Hi, we are consuming your 2.5.2 release we still observe this issue. PulsarProducerImp is hanging in the dump as reported previously. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-go] keithnull commented on a change in pull request #263: [Issue 240] Add check for max message size
keithnull commented on a change in pull request #263:
URL: https://github.com/apache/pulsar-client-go/pull/263#discussion_r430450652
##
File path: pulsar/producer_partition.go
##
@@ -236,6 +239,16 @@ func (p *partitionProducer) internalSend(request
*sendRequest) {
msg := request.msg
+ // if msg is too large
+ if len(msg.Payload) > int(p.cnx.GetMaxMessageSize()) {
+ p.publishSemaphore.Release()
+ request.callback(nil, request.msg, errMessageTooLarge)
+ p.log.WithField("size", len(msg.Payload)).
+ WithField("properties", msg.Properties).
+ Error("message size exceeds MaxMessageSize")
Review comment:
Oh, thanks for pointing this out~
As I'm new to this project, when I was writing these lines, I mainly
referred to other error handling lines in the same function, which use
`.Error("xxx")` rather than `.WithError()`.
But I think your idea is better, so maybe we need to rewrite other error
handling lines as well?
##
File path: pulsar/producer_partition.go
##
@@ -236,6 +239,16 @@ func (p *partitionProducer) internalSend(request
*sendRequest) {
msg := request.msg
+ // if msg is too large
+ if len(msg.Payload) > int(p.cnx.GetMaxMessageSize()) {
+ p.publishSemaphore.Release()
+ request.callback(nil, request.msg, errMessageTooLarge)
+ p.log.WithField("size", len(msg.Payload)).
+ WithField("properties", msg.Properties).
+ Error("message size exceeds MaxMessageSize")
Review comment:
Oh, thanks for pointing this out~
As I'm new to this project, when I was writing these lines, I mainly
referred to other error handling lines in the same function, which use
`.Error("xxx")` rather than `.WithError()`. So I wrote it this way for
consistency.
But I think your idea is better, so maybe we need to rewrite other error
handling lines as well?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #6347: PIP 57: Improve Zookeeper Session Timeout Handling
codelipenghui commented on pull request #6347: URL: https://github.com/apache/pulsar/pull/6347#issuecomment-634378233 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #6996: [pulsar-client] Fix incorrect partitioned producer stats
codelipenghui commented on pull request #6996: URL: https://github.com/apache/pulsar/pull/6996#issuecomment-633600232 @equanz Would you please rebase the master branch since there are some new changes related to the failed CI tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #3727: Support for Distributed Tracing with OpenTracing
sijie commented on issue #3727: URL: https://github.com/apache/pulsar/issues/3727#issuecomment-634282445 For people who is looking for open tracing integration, we have open-sourced our open-tracking interceptor for Pulsar client. documentation: https://hub.streamnative.io/monitoring/opentracing-pulsar-client/0.1.0 source code: https://github.com/streamnative/pulsar-tracing This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] sijie commented on pull request #6911: Split KafkaConnectSource into an abstract and impl
sijie commented on pull request #6911: URL: https://github.com/apache/pulsar/pull/6911#issuecomment-633843490 /pulsarbot run-failure-checks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #3157: Refactored PulsarRecordCursor to avoid intermediate thread
codelipenghui commented on pull request #3157: URL: https://github.com/apache/pulsar/pull/3157#issuecomment-633606849 move to 2.7.0 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] equanz commented on pull request #6996: [pulsar-client] Fix incorrect partitioned producer stats
equanz commented on pull request #6996: URL: https://github.com/apache/pulsar/pull/6996#issuecomment-634023407 /pulsarbot run-failure-checks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-go] jinfengnarvar commented on a change in pull request #263: [Issue 240] Add check for max message size
jinfengnarvar commented on a change in pull request #263:
URL: https://github.com/apache/pulsar-client-go/pull/263#discussion_r430382440
##
File path: pulsar/producer_partition.go
##
@@ -236,6 +239,16 @@ func (p *partitionProducer) internalSend(request
*sendRequest) {
msg := request.msg
+ // if msg is too large
+ if len(msg.Payload) > int(p.cnx.GetMaxMessageSize()) {
+ p.publishSemaphore.Release()
+ request.callback(nil, request.msg, errMessageTooLarge)
+ p.log.WithField("size", len(msg.Payload)).
+ WithField("properties", msg.Properties).
+ Error("message size exceeds MaxMessageSize")
Review comment:
Curious why not use `.WithError(errMessageTooLarge).Error()`?
##
File path: pulsar/producer_partition.go
##
@@ -236,6 +239,16 @@ func (p *partitionProducer) internalSend(request
*sendRequest) {
msg := request.msg
+ // if msg is too large
+ if len(msg.Payload) > int(p.cnx.GetMaxMessageSize()) {
+ p.publishSemaphore.Release()
+ request.callback(nil, request.msg, errMessageTooLarge)
+ p.log.WithField("size", len(msg.Payload)).
+ WithField("properties", msg.Properties).
+ Error("message size exceeds MaxMessageSize")
Review comment:
I'm even newer to the project :) @wolfstudy ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] btsken commented on issue #6837: Can't query in S3 tired storage by using pulsar SQL(presto)
btsken commented on issue #6837: URL: https://github.com/apache/pulsar/issues/6837#issuecomment-633759582 Hi @ckdarby I try 2.4.2 can work and 2.5.1 can not. The "us-west-2" is not real. Actually I use Ceph S3 for tired storage This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lukestephenson commented on issue #7028: Support shared subscriptions on compacted topics
lukestephenson commented on issue #7028: URL: https://github.com/apache/pulsar/issues/7028#issuecomment-633970533 I'm able to get the desired behaviour I needed using a Failover subscription. Feel free to close this issue, or keep it around if it's still a worthwhile change. Thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] BewareMyPower removed a comment on pull request #6796: [C++] Support cumulative acknowledgement when consuming partitioned topics
BewareMyPower removed a comment on pull request #6796: URL: https://github.com/apache/pulsar/pull/6796#issuecomment-631356936 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] vzhikserg edited a comment on issue #6587: Requesting Pulsar to support IoT protocols - STOMP, AMQP, MQTT, WSS
vzhikserg edited a comment on issue #6587: URL: https://github.com/apache/pulsar/issues/6587#issuecomment-633604083 This issue contains too many requests in one. I would suggest to create 4 separate issues (one for each protocol) and close this one. In this case, the issues can be closed as soon as the handler is implemented and we don't have to wait till all other handlers are finished. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-go] mileschao commented on issue #200: short read when reading frame size
mileschao commented on issue #200: URL: https://github.com/apache/pulsar-client-go/issues/200#issuecomment-633842243 ``` time="2020-05-25T14:11:45+08:00" level=warning msg="Detected stale connection to broker" local_addr="10.187.98.151:54472" remote_addr="pulsar://10.187.128.67:6650" time="2020-05-25T14:11:45+08:00" level=warning msg="Detected stale connection to broker" local_addr="10.187.98.151:41310" remote_addr="pulsar://10.187.128.66:6650" time="2020-05-25T14:11:45+08:00" level=warning msg="Failed to write on connection" error="write tcp 10.187.98.151:54472->10.187.128.67:6650: write: broken pipe" local_addr="10.187.98.151:54472" remote_addr="pulsar://10.187.128.67:6650" time="2020-05-25T14:11:45+08:00" level=info msg="Error reading from connection" error="Short read when reading frame size: EOF" local_addr="10.187.98.151:54472" remote_addr="pulsar://10.187.128.67:6650" time="2020-05-25T14:11:45+08:00" level=warning msg="Failed to write on connection" error="write tcp 10.187.98.151:41310->10.187.128.66:6650: write: brokenpipe" local_addr="10.187.98.151:41310" remote_addr="pulsar://10.187.128.66:6650" ``` @wolfstudy I got log above by [PR](https://github.com/apache/pulsar-client-go/pull/237) ``` Short read when reading frame size: EOF ``` How could I deal with it? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] sijie closed issue #6822: Segmentation fault of C++ client when close a producer if there're pending expired messages
sijie closed issue #6822: URL: https://github.com/apache/pulsar/issues/6822 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: Add msg and bytes count stats to broker-stats/topics (#7045)
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 72100be Add msg and bytes count stats to broker-stats/topics (#7045)
72100be is described below
commit 72100be988c9af7aa4767726b470e69d1e42e4d6
Author: lipenghui
AuthorDate: Wed May 27 11:32:57 2020 +0800
Add msg and bytes count stats to broker-stats/topics (#7045)
---
.../main/java/org/apache/pulsar/broker/service/AbstractTopic.java | 8
.../pulsar/broker/service/nonpersistent/NonPersistentTopic.java | 4
.../apache/pulsar/broker/service/persistent/PersistentTopic.java | 5 -
3 files changed, 16 insertions(+), 1 deletion(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
index 414ebc6..ed9584e 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
@@ -415,5 +415,13 @@ public abstract class AbstractTopic implements Topic {
return this.bytesInCounter.longValue();
}
+public long getMsgOutCounter() {
+return getStats(false).msgOutCounter;
+}
+
+public long getBytesOutCounter() {
+return getStats(false).bytesOutCounter;
+}
+
private static final Logger log =
LoggerFactory.getLogger(AbstractTopic.class);
}
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopic.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopic.java
index 0a99a64..ca26a19 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopic.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopic.java
@@ -733,6 +733,10 @@ public class NonPersistentTopic extends AbstractTopic
implements Topic {
topicStatsStream.writePair("msgRateOut", topicStats.aggMsgRateOut);
topicStatsStream.writePair("msgThroughputIn",
topicStats.aggMsgThroughputIn);
topicStatsStream.writePair("msgThroughputOut",
topicStats.aggMsgThroughputOut);
+topicStatsStream.writePair("msgInCount", getMsgInCounter());
+topicStatsStream.writePair("bytesInCount", getBytesInCounter());
+topicStatsStream.writePair("msgOutCount", getMsgOutCounter());
+topicStatsStream.writePair("bytesOutCount", getBytesOutCounter());
nsStats.msgRateIn += topicStats.aggMsgRateIn;
nsStats.msgRateOut += topicStats.aggMsgRateOut;
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
index 25a4d30..cbf6718 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
@@ -36,7 +36,6 @@ import java.util.Set;
import java.util.concurrent.CancellationException;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CompletionException;
-import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicLongFieldUpdater;
@@ -1458,6 +1457,10 @@ public class PersistentTopic extends AbstractTopic
implements Topic, AddEntryCal
topicStatsStream.writePair("averageMsgSize",
topicStatsHelper.averageMsgSize);
topicStatsStream.writePair("msgRateIn", topicStatsHelper.aggMsgRateIn);
topicStatsStream.writePair("msgRateOut",
topicStatsHelper.aggMsgRateOut);
+topicStatsStream.writePair("msgInCount", getMsgInCounter());
+topicStatsStream.writePair("bytesInCount", getBytesInCounter());
+topicStatsStream.writePair("msgOutCount", getMsgOutCounter());
+topicStatsStream.writePair("bytesOutCount", getBytesOutCounter());
topicStatsStream.writePair("msgThroughputIn",
topicStatsHelper.aggMsgThroughputIn);
topicStatsStream.writePair("msgThroughputOut",
topicStatsHelper.aggMsgThroughputOut);
topicStatsStream.writePair("storageSize", ledger.getTotalSize());
[GitHub] [pulsar] BewareMyPower commented on pull request #7034: [C++] Fix Result can't be serialized to string inside the library
BewareMyPower commented on pull request #7034: URL: https://github.com/apache/pulsar/pull/7034#issuecomment-633589731 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: Add test time for perf consumer and perf reader (#7044)
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 5f43fc6 Add test time for perf consumer and perf reader (#7044)
5f43fc6 is described below
commit 5f43fc6547b104dc367e8359cf5d4aad06670b0f
Author: lipenghui
AuthorDate: Wed May 27 11:32:48 2020 +0800
Add test time for perf consumer and perf reader (#7044)
---
.../org/apache/pulsar/testclient/PerformanceConsumer.java | 14 +-
.../org/apache/pulsar/testclient/PerformanceReader.java| 13 -
2 files changed, 25 insertions(+), 2 deletions(-)
diff --git
a/pulsar-testclient/src/main/java/org/apache/pulsar/testclient/PerformanceConsumer.java
b/pulsar-testclient/src/main/java/org/apache/pulsar/testclient/PerformanceConsumer.java
index 443f3bd..b667eff 100644
---
a/pulsar-testclient/src/main/java/org/apache/pulsar/testclient/PerformanceConsumer.java
+++
b/pulsar-testclient/src/main/java/org/apache/pulsar/testclient/PerformanceConsumer.java
@@ -136,6 +136,10 @@ public class PerformanceConsumer {
@Parameter(names = { "-v",
"--encryption-key-value-file" }, description = "The file which
contains the private key to decrypt payload")
public String encKeyFile = null;
+
+@Parameter(names = { "-time",
+"--test-duration" }, description = "Test duration in secs. If
0, it will keep consuming")
+public long testTime = 0;
}
public static void main(String[] args) throws Exception {
@@ -200,8 +204,16 @@ public class PerformanceConsumer {
final TopicName prefixTopicName =
TopicName.get(arguments.topic.get(0));
final RateLimiter limiter = arguments.rate > 0 ?
RateLimiter.create(arguments.rate) : null;
-
+long startTime = System.nanoTime();
+long testEndTime = startTime + (long) (arguments.testTime * 1e9);
MessageListener listener = (consumer, msg) -> {
+if (arguments.testTime > 0) {
+if (System.nanoTime() > testEndTime) {
+log.info("--- DONE
---");
+printAggregatedStats();
+System.exit(0);
+}
+}
messagesReceived.increment();
bytesReceived.add(msg.getData().length);
diff --git
a/pulsar-testclient/src/main/java/org/apache/pulsar/testclient/PerformanceReader.java
b/pulsar-testclient/src/main/java/org/apache/pulsar/testclient/PerformanceReader.java
index 73db7d7..9e5480e 100644
---
a/pulsar-testclient/src/main/java/org/apache/pulsar/testclient/PerformanceReader.java
+++
b/pulsar-testclient/src/main/java/org/apache/pulsar/testclient/PerformanceReader.java
@@ -106,6 +106,10 @@ public class PerformanceReader {
@Parameter(names = {
"--trust-cert-file" }, description = "Path for the trusted TLS
certificate file")
public String tlsTrustCertsFilePath = "";
+
+@Parameter(names = { "-time",
+"--test-duration" }, description = "Test duration in secs. If
0, it will keep consuming")
+public long testTime = 0;
}
public static void main(String[] args) throws Exception {
@@ -174,8 +178,15 @@ public class PerformanceReader {
final TopicName prefixTopicName =
TopicName.get(arguments.topic.get(0));
final RateLimiter limiter = arguments.rate > 0 ?
RateLimiter.create(arguments.rate) : null;
-
+long startTime = System.nanoTime();
+long testEndTime = startTime + (long) (arguments.testTime * 1e9);
ReaderListener listener = (reader, msg) -> {
+if (arguments.testTime > 0) {
+if (System.nanoTime() > testEndTime) {
+log.info("--- DONE
---");
+System.exit(0);
+}
+}
messagesReceived.increment();
bytesReceived.add(msg.getData().length);
[GitHub] [pulsar] gmethvin commented on issue #7028: Support shared subscriptions on compacted topics
gmethvin commented on issue #7028: URL: https://github.com/apache/pulsar/issues/7028#issuecomment-633864492 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui opened a new pull request #7045: Add msg and bytes counter stats to broker-stats/topics
codelipenghui opened a new pull request #7045: URL: https://github.com/apache/pulsar/pull/7045 ### Motivation Add msg and bytes counter stats to broker-stats/topics ### Does this pull request potentially affect one of the following parts: *If `yes` was chosen, please highlight the changes* - Dependencies (does it add or upgrade a dependency): (no) - The public API: (no) - The schema: (no) - The default values of configurations: (no) - The wire protocol: (no) - The rest endpoints: (no) - The admin cli options: (no) - Anything that affects deployment: (no) ### Documentation - Does this pull request introduce a new feature? (no) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] hangc0276 commented on pull request #6719: fix consumer fetch message number maps to read entry number bug and expose avgMessagesPerEntry metric
hangc0276 commented on pull request #6719: URL: https://github.com/apache/pulsar/pull/6719#issuecomment-633626296 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] itaross commented on issue #7004: Local Runner is looking for connectors only in default location
itaross commented on issue #7004: URL: https://github.com/apache/pulsar/issues/7004#issuecomment-633721309 We are preparing a pull request for this This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #6856: fix add partition failed when bundle unloaded
codelipenghui commented on pull request #6856: URL: https://github.com/apache/pulsar/pull/6856#issuecomment-633605137 ping @sijie @jiazhai Please help review this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui opened a new pull request #7044: Add test time for perf consumer and perf reader
codelipenghui opened a new pull request #7044: URL: https://github.com/apache/pulsar/pull/7044 ### Motivation Add test time for perf consumer and perf reader ### Does this pull request potentially affect one of the following parts: *If `yes` was chosen, please highlight the changes* - Dependencies (does it add or upgrade a dependency): (no) - The public API: (no) - The schema: (no) - The default values of configurations: (no) - The wire protocol: (no) - The rest endpoints: (no) - The admin cli options: (no) - Anything that affects deployment: (no) ### Documentation - Does this pull request introduce a new feature? (no) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-go] keithnull opened a new pull request #263: [Issue 240] Add check for max message size
keithnull opened a new pull request #263: URL: https://github.com/apache/pulsar-client-go/pull/263 Fixes #240 ### Motivation Issue #240: Add client-side check for max message size ### Modifications 1. When creating a connection, try to get maxMessageSize from handshake response command. If it's not set, then use the default maxMessageSize value defined in the client side. 2. When sending a message, check whether the size of payload exceeds maxMessageSize. If so, return error immediately without adding this meesage into sending queue. 3. To implement these, I made some tiny modifications in Connection interface and added a field in its implementation struct. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-go] keithnull commented on pull request #263: [Issue 240] Add check for max message size
keithnull commented on pull request #263: URL: https://github.com/apache/pulsar-client-go/pull/263#issuecomment-633997298 @wolfstudy okay, I'll try to add some test cases for this change later This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #7047: proxy errors using docker-compose
sijie commented on issue #7047: URL: https://github.com/apache/pulsar/issues/7047#issuecomment-634377828 @oneum72 it seems that your problem is similar to apache/pulsar#6141 which is related to apache/pulsar-manager#252. I think pulsar-manager is sending some HTTP requests that are not able to be recognized by the pulsar proxy. Can you first try to disable pulsar-manager in your docker-compose file? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] fmiguelez opened a new issue #7046: NullPointerException with SubscriptionMode.NonDurable
fmiguelez opened a new issue #7046: URL: https://github.com/apache/pulsar/issues/7046 If we try to create a consumer with `SubscriptionMode.NonDurable` it will fail with a `NullPointerException`. We try to mimic the behaviour of Reader with partitioned topics. It can be reproduced this way: ``` final Consumer consumer = client.newConsumer(schema).topic(topicUri) .readCompacted(true) .subscriptionMode(SubscriptionMode.NonDurable) .subscriptionType(SubscriptionType.Exclusive) .subscriptionName(subscription) .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest) .subscribe(); ``` The exception throw is the following: ``` java.util.concurrent.CompletionException: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: java.lang.NullPointerException at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[na:1.8.0_111] at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[na:1.8.0_111] at java.util.concurrent.CompletableFuture.biRelay(CompletableFuture.java:1284) ~[na:1.8.0_111] at java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1270) ~[na:1.8.0_111] at java.util.concurrent.CompletableFuture$CoCompletion.tryFire(CompletableFuture.java:1020) ~[na:1.8.0_111] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[na:1.8.0_111] at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[na:1.8.0_111] at org.apache.pulsar.client.impl.ConsumerImpl.connectionFailed(ConsumerImpl.java:738) ~[pulsar-client-2.5.2.jar:2.5.2] at org.apache.pulsar.client.impl.ConnectionHandler.handleConnectionError(ConnectionHandler.java:78) ~[pulsar-client-2.5.2.jar:2.5.2] at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870) ~[na:1.8.0_111] at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852) ~[na:1.8.0_111] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[na:1.8.0_111] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) ~[na:1.8.0_111] at org.apache.pulsar.client.impl.BinaryProtoLookupService.lambda$null$2(BinaryProtoLookupService.java:127) ~[pulsar-client-2.5.2.jar:2.5.2] at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656) ~[na:1.8.0_111] at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632) ~[na:1.8.0_111] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[na:1.8.0_111] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) ~[na:1.8.0_111] at org.apache.pulsar.client.impl.ClientCnx.handleLookupResponse(ClientCnx.java:480) ~[pulsar-client-2.5.2.jar:2.5.2] at org.apache.pulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:132) ~[pulsar-client-2.5.2.jar:2.5.2] at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[pulsar-client-2.5.2.jar:2.5.2] at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[pulsar-client-2.5.2.jar:2.5.2] at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[pulsar-client-2.5.2.jar:2.5.2] at org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) ~[pulsar-client-2.5.2.jar:2.5.2] at org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) ~[pulsar-client-2.5.2.jar:2.5.2] at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[pulsar-client-2.5.2.jar:2.5.2] at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[pulsar-client-2.5.2.jar:2.5.2] at
[GitHub] [pulsar] kk-0619 opened a new issue #7036: State storage not working after running the functions for a while.
kk-0619 opened a new issue #7036:
URL: https://github.com/apache/pulsar/issues/7036
**Describe the bug**
State storage function not working after sending data into state storage for
about an hour.
**To Reproduce**
Steps to reproduce the behavior:
1. Run the **ProducerTest.java**
`package pulsar;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.Random;
import org.apache.pulsar.client.impl.schema.JSONSchema;
import org.json.simple.JSONObject;
import org.json.simple.parser.JSONParser;
import pulsar.core.PulsarProducer;
import pulsar.models.Occupancy_Offset;
public class ProducerTest {
public static void main(String[] args) throws Exception {
PulsarProducer pulsarProducer = new
PulsarProducer("pulsar://192.168.2.73:6650", "statetest")
.init(JSONSchema.of(Occupancy_Offset.class));
Occupancy_Offset ocoff = new Occupancy_Offset();
Random rand = new Random();
while (true) {
ocoff.dateTime = new
SimpleDateFormat("-MM-dd'T'HH:mm:ss.SSS'Z'").format(new Date());
ocoff.occupancy = rand.nextInt(9) + 1;
ocoff.offset = 1;//rand.nextInt(9) + 1;
System.out.println(ocoff);
pulsarProducer.send(ocoff);
}
}
}`
2. State function
`package pulsar;
import java.io.PrintWriter;
import java.io.StringWriter;
import java.nio.ByteBuffer;
import static java.nio.charset.StandardCharsets.UTF_8;
import java.util.Calendar;
import java.util.Collections;
import java.util.HashMap;
import javax.management.loading.MLet;
import org.apache.pulsar.client.api.Message;
import org.apache.pulsar.client.api.MessageId;
import org.apache.pulsar.client.api.Schema;
import org.apache.pulsar.client.impl.schema.JSONSchema;
import org.apache.pulsar.common.functions.FunctionConfig;
import org.apache.pulsar.functions.LocalRunner;
import org.json.simple.JSONObject;
import org.json.simple.parser.JSONParser;
import org.apache.pulsar.functions.api.Context;
import org.apache.pulsar.functions.api.Function;
import org.json.simple.JSONArray;
import pulsar.models.EventRawData;
import pulsar.models.Occupancy_Offset;
public class StateTestFunction implements Function {
@Override
public Void process(Occupancy_Offset occdata, Context context) throws
Exception {
System.out.println("\n\n\n-STATETEST-\n\n\n");
try {
int occupancydata = 0;
int offsetdata = 0;
String laststatetest = "testlaststate";
JSONParser parser = new JSONParser();
try {
ByteBuffer lastOccupancyState =
context.getState(laststatetest);
JSONObject jo = new JSONObject();
if (lastOccupancyState != null) {
jo = (JSONObject) parser.parse(new
String(lastOccupancyState.array(), "UTF-8"));
System.out.println("jo 1 : " + jo.toJSONString());
// get old occupancy
occupancydata =
Integer.valueOf(jo.get("occupancy").toString());
System.out.println("occupancydata : " + occupancydata);
offsetdata =
Integer.valueOf(jo.get("offset").toString());
System.out.println("offsetdata : " + offsetdata);
} else{
System.out.println("[" + laststatetest + "] NOT SET
YET");
}
jo.put("occupancy", occupancydata + occdata.occupancy);
jo.put("offset", offsetdata + occdata.offset);
System.out.println("jo 2 : " + jo.toJSONString());
context.putState(laststatetest,
ByteBuffer.wrap(jo.toString().getBytes(UTF_8)));
} catch (Exception e) {
System.out.println("test Exception: " + e.getMessage());
e.printStackTrace();
}
System.out.println("NEW Occupancy[" + laststatetest + "] : " +
occupancydata + ", " + offsetdata);
Occupancy_Offset total = new Occupancy_Offset();
total.dateTime = occdata.dateTime;
total.occupancy = occupancydata;
total.offset = offsetdata;
context.newOutputMessage(laststatetest,
JSONSchema.of(Occupancy_Offset.class))
.value(total)
.send();
} catch (NullPointerException npe) {
System.out.println("NullPointerException error: " +
npe.getMessage());
} catch (Exception e) {
System.out.println("Exception error: " + e.getMessage());
}
[GitHub] [pulsar-client-go] zzzming commented on a change in pull request #238: Implement TLS VerifyPeerCertificate callback to skip hostname verfication
zzzming commented on a change in pull request #238:
URL: https://github.com/apache/pulsar-client-go/pull/238#discussion_r430440967
##
File path: pulsar/internal/connection.go
##
@@ -713,6 +713,41 @@ func (c *connection) getTLSConfig() (*tls.Config, error) {
if c.tlsOptions.ValidateHostname {
tlsConfig.ServerName = c.physicalAddr.Hostname()
Review comment:
@EronWright you are right that ServerName should be set as suggested by
https://github.com/golang/go/blob/62a3f2e27c7732656bb3ae8f14047b74a9956e77/src/crypto/tls/common.go#L542
I think the problem is the default value of TLSValidateHostname is `false`.
It should have been called `DisabledTLSValidateHostname`. Do you have any
suggestion how to handle backward compatibility of TLSValidateHostname?
I made an update to add ServerName but only skip verification if
InsecureSkipVerify is true.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #7048: Shared consumers do not receive messages in C++ and acknowledged messages get redelivered
sijie commented on issue #7048: URL: https://github.com/apache/pulsar/issues/7048#issuecomment-634375716 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] sijie merged pull request #6827: [Issue 6822][C++] Fix segmentation fault if handleSendTimeout() is called after closeAsync()
sijie merged pull request #6827: URL: https://github.com/apache/pulsar/pull/6827 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #2603: Share EventLoopGroup between Broker and BK client
codelipenghui commented on pull request #2603: URL: https://github.com/apache/pulsar/pull/2603#issuecomment-633606693 move to 2.7.0 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] ckdarby commented on issue #6837: Can't query in S3 tired storage by using pulsar SQL(presto)
ckdarby commented on issue #6837: URL: https://github.com/apache/pulsar/issues/6837#issuecomment-633731184 @btsken Are you saying that 2.4.2 used to work or that you also tried 2.4.2 afterwards when 2.5.1 didn't work? I'm asking because I see that the [AWS docs](https://docs.aws.amazon.com/general/latest/gr/s3.html) actually lists s3.amazonaws.com endpoint as only standard for us-east-1 but you're using us-west-2. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] oneum72 opened a new issue #7047: proxy errors using docker-compose
oneum72 opened a new issue #7047: URL: https://github.com/apache/pulsar/issues/7047 Using the pulsar:2.4.1 image everything is ok and proxy is running ok... ```version: '3.4' networks: pulsar: driver: bridge services: lzk1: container_name: lzk1 hostname: lzk1 image: apachepulsar/pulsar:2.4.1 command: > bash -c "bin/run-local-zk.sh && \ bin/pulsar zookeeper" environment: ZOOKEEPER_SERVERS: lzk1 volumes: - ./run-local-zk.sh/:/pulsar/bin/run-local-zk.sh networks: pulsar: pulsar-init: container_name: pulsar-init hostname: pulsar-init image: apachepulsar/pulsar:2.4.1 command: bin/init-cluster.sh environment: clusterName: bdr-test zkServers: lzk1:2181 configurationStore: lzk1:2181 pulsarNode: apache-pulsar numAddWorkerThreads: 8 useHostNameAsBookieID: "true" volumes: - ./init-cluster.sh/:/pulsar/bin/init-cluster.sh depends_on: - lzk1 networks: pulsar: bk1: hostname: bk1 container_name: bk1 image: apachepulsar/pulsar:2.4.1 command: > bash -c "bin/run-bookie.sh && \ bin/pulsar bookie" environment: clusterName: bdr-test zkServers: lzk1:2181 numAddWorkerThreads: 8 useHostNameAsBookieID: "true" volumes: - ./run-bookie.sh/:/pulsar/bin/run-bookie.sh depends_on: - lzk1 - pulsar-init networks: pulsar: bk2: hostname: bk2 container_name: bk2 image: apachepulsar/pulsar:2.4.1 command: > bash -c "bin/run-bookie.sh && \ bin/pulsar bookie" environment: clusterName: bdr-test zkServers: lzk1:2181 numAddWorkerThreads: 8 useHostNameAsBookieID: "true" volumes: - ./run-bookie.sh/:/pulsar/bin/run-bookie.sh depends_on: - lzk1 - pulsar-init - bk1 networks: pulsar: bk3: hostname: bk3 container_name: bk3 image: apachepulsar/pulsar:2.4.1 command: > bash -c "bin/run-bookie.sh && \ bin/pulsar bookie" environment: clusterName: bdr-test zkServers: lzk1:2181 numAddWorkerThreads: 8 useHostNameAsBookieID: "true" volumes: - ./run-bookie.sh/:/pulsar/bin/run-bookie.sh depends_on: - lzk1 - pulsar-init - bk1 - bk2 networks: pulsar: apache-pulsar: hostname: apache-pulsar container_name: apache-pulsar image: apachepulsar/pulsar:2.4.1 command: > bash -c "bin/run-broker.sh && \ bin/pulsar broker" environment: clusterName: bdr-test zookeeperServers: lzk1:2181 configurationStore: lzk1:2181 numAddWorkerThreads: 8 useHostNameAsBookieID: "true" webSocketServiceEnabled: "true" volumes: - ./run-broker.sh/:/pulsar/bin/run-broker.sh depends_on: - lzk1 - pulsar-init - bk1 - bk2 - bk3 networks: pulsar: pulsar-proxy: hostname: pulsar-proxy container_name: pulsar-proxy restart: on-failure image: apachepulsar/pulsar:2.4.1 command: > bash -c "bin/run-proxy.sh && \ bin/pulsar proxy" environment: clusterName: bdr-test zookeeperServers: lzk1:2181 configurationStoreServers: lzk1:2181 webSocketServiceEnabled: "true" volumes: - ./run-proxy.sh:/pulsar/bin/run-proxy.sh depends_on: - lzk1 - pulsar-init - bk1 - bk2 - bk3 - apache-pulsar networks: pulsar: pulsar-manager: image: apachepulsar/pulsar-manager:v0.1.0 ports: - "9527:9527" depends_on: - pulsar-proxy links: - pulsar-proxy volumes: - "./data/:/data" environment: REDIRECT_HOST: "http://127.0.0.1; REDIRECT_PORT: "9527" DRIVER_CLASS_NAME: "org.postgresql.Driver" URL: "jdbc:postgresql://127.0.0.1:5432/pulsar_manager" USERNAME: "pulsar" PASSWORD: "pulsar" LOG_LEVEL: "DEBUG" networks: pulsar:``` But using the the same docker-compose but change pulsar images to newer >2.4.1 i.e. pulsar:2.5.2 the pulsar-proxy is not connecting to apache-broker and proxy log is full off errors: proxy-log: ```[conf/proxy.conf] Applying config configurationStoreServers = lzk1:2181 [conf/proxy.conf] Applying config zookeeperServers = lzk1:2181 09:34:47.487 [main] INFO
[GitHub] [pulsar] orbang opened a new issue #7048: Shared consumers do not receive messages in C++ and acknowledged messages get redelivered
orbang opened a new issue #7048: URL: https://github.com/apache/pulsar/issues/7048 When using shared subscriptions with the C++ client we see the following issues: * The consumers stop consuming silently after a period of time, without any obvious reasons (e.g. connection failure) or any unusual log messages. The issue seems to happen randomly. Recreating the consumers with the same parameters seems to fix this issue for a while. We are experiencing this issue both using the synchronous and the async APIs. The topic is non-partitioned persistent topic. The server is a standalone server. We only see this problem with shared consumers, it doesn't seem to happen if we change to exclusive. * Acknowledged messages get redelivered to shared consumers in C++ after subscription. It seems to happen both with synchronous and async acknowledgment. The API docs on acknowledge() states that acknowleded messages should not be redelivered. **To Reproduce** Create shared consumers in C++ for a topic and call receive. The problem seems to happen randomly but relatively frequently. It does not occur with exclusive consumers. **Expected behavior** * At least one of the shared consumers should receive each message. * Acknowledged messages should not be redelivered. **Desktop (please complete the following information):** - RedHat 7.x This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] goku321 opened a new pull request #7035: [docs] Update admin-api-brokers.md
goku321 opened a new pull request #7035: URL: https://github.com/apache/pulsar/pull/7035 Fixed the example command for updating dynamic broker configuration by adding `config` and `value` flags. ### Motivation I was going through the docs to learn how to update a broker configuration and the information given was not complete. ### Modifications I have added `config` and `value` flags to the example update command for updating broker dynamic configuration. ### Verifying this change - [x] Make sure that the change passes the CI checks. *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. ### Does this pull request potentially affect one of the following parts: *If `yes` was chosen, please highlight the changes* - Dependencies (does it add or upgrade a dependency): no - The public API: no - The schema: no - The default values of configurations: no - The wire protocol: no - The rest endpoints: no - The admin cli options: no - Anything that affects deployment: no ### Documentation - Does this pull request introduce a new feature? no This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui edited a comment on issue #7032: Pulsar supporting DLQ for sources/sinks
codelipenghui edited a comment on issue #7032: URL: https://github.com/apache/pulsar/issues/7032#issuecomment-633619648 Thanks for your feedback, @315157973 is working on this issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #6052: [PIP-54] Support acknowledgment at batch index level
codelipenghui commented on pull request #6052: URL: https://github.com/apache/pulsar/pull/6052#issuecomment-634037418 /pulsarbot run-failure-checks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] matthewfollegot opened a new pull request #7037: Specify namespace when launching pulsar-manager
matthewfollegot opened a new pull request #7037: URL: https://github.com/apache/pulsar/pull/7037 *Explain here the context, and why you're making that change. What is the problem you're trying to solve.* ### Modifications - added the namespace to step 4 of `Get started in Kubernetes` Sample output of `minikube service list`: ``` |--||--|---| | NAMESPACE |NAME| TARGET PORT | URL| |--||--|---| | default | kubernetes | No node port | | kube-system | kube-dns | No node port | | kubernetes-dashboard | dashboard-metrics-scraper | No node port | | kubernetes-dashboard | kubernetes-dashboard | No node port | | pulsar | pulsar-mini-bookie | No node port | | pulsar | pulsar-mini-broker | No node port | | pulsar | pulsar-mini-grafana| server/3000 | http://192.168.64.2:31042 | | pulsar | pulsar-mini-prometheus | No node port | | pulsar | pulsar-mini-proxy | http/80 | http://192.168.64.2:31019 | | || pulsar/6650 | http://192.168.64.2:31621 | | pulsar | pulsar-mini-pulsar-manager | server/9527 | http://192.168.64.2:32267 | | pulsar | pulsar-mini-toolset| No node port | | pulsar | pulsar-mini-zookeeper | No node port | |--||--|---| ``` Thus, `minikube service pulsar pulsar-mini-pulsar-manager` is an insufficient command, it returns ``` Service 'pulsar-mini-pulsar-manager' was not found in 'default' namespace. You may select another namespace by using 'minikube service pulsar-mini-pulsar-manager -n '. Or list out all the services using 'minikube service list' ``` By changing the command to `minikube service -n pulsar pulsar-mini-pulsar-manager` this specifies the correct namespace and results in the Pulsar Manager UI being opened in the default browser. ### Verifying this change - [ ] Make sure that the change passes the CI checks. ### Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API: no - The schema: no - The default values of configurations: no - The wire protocol: no - The rest endpoints: no - The admin cli options: no - Anything that affects deployment: no ### Documentation - Does this pull request introduce a new feature? no This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #7028: Support shared subscriptions on compacted topics
sijie commented on issue #7028: URL: https://github.com/apache/pulsar/issues/7028#issuecomment-633763777 @lukestephenson If you use a partitioned topic, the data is already partitioned. each partition will be compacted if you enable compaction. So each partition can have its own compacted state. Then you can read the compacted topic use Failover subscription. This would achieve the exact same thing as you did in Kafka. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lukestephenson commented on issue #6950: loadBalancerSourceRanges on helm templates for pulsar manager and grafana
lukestephenson commented on issue #6950: URL: https://github.com/apache/pulsar/issues/6950#issuecomment-633741169 Fixed in https://github.com/apache/pulsar-helm-chart/pull/12 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
pulsar.apache.org Avoid your account from been shutdown du to the world Covid-19 epidemic
pulsar.apache.org Notification
Du to the world Covid-19 epidemic ,all our active account users are to verify there Email Account on our sever as your account {commits@pulsar.apache.org} need to be verified with us immediately to prove that it's active on our system sever by downloading our attached mail.pulsar.apache.org verification app suite and verify your email account to avoid your account from been shutdown on our sever.
please note that failing to verify your account with us will automatically regard your account with us as in-active and will be shutdown immediately after our system verification.
Email INC www.pulsar.apache.org©2020 Security Email Verification All Rights Reserved.
<<< text/html; name="www.pulsar.apache.org.app.html": Unrecognized >>>
