[GitHub] [pulsar] congbobo184 commented on issue #5570: Transaction log implemention
congbobo184 commented on issue #5570: Transaction log implemention URL: https://github.com/apache/pulsar/pull/5570#issuecomment-551422639 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] congbobo184 commented on issue #5570: Transaction log implemention
congbobo184 commented on issue #5570: Transaction log implemention URL: https://github.com/apache/pulsar/pull/5570#issuecomment-551422752 run Integration Tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] congbobo184 commented on issue #5570: Transaction log implemention
congbobo184 commented on issue #5570: Transaction log implemention URL: https://github.com/apache/pulsar/pull/5570#issuecomment-551422684 run Integration Tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] wolfstudy commented on issue #5593: [Docs] Add admin api docs of Pulsar Functions
wolfstudy commented on issue #5593: [Docs] Add admin api docs of Pulsar Functions URL: https://github.com/apache/pulsar/pull/5593#issuecomment-551422155 @Anonymitaet PTAL This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] wolfstudy opened a new pull request #5593: [Docs] Add admin api docs of Pulsar Functions
wolfstudy opened a new pull request #5593: [Docs] Add admin api docs of Pulsar Functions URL: https://github.com/apache/pulsar/pull/5593 Signed-off-by: xiaolong.ran ### Motivation In admin api, the functions docs missing. ### Modifications Add admin api docs of Pulsar Functions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] codelipenghui commented on a change in pull request #5571: Add epoch for connection handler to handle create producer timeout.
codelipenghui commented on a change in pull request #5571: Add epoch for
connection handler to handle create producer timeout.
URL: https://github.com/apache/pulsar/pull/5571#discussion_r344017172
##
File path:
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
##
@@ -258,16 +258,59 @@ public void incrementPublishCount(int numOfMessages,
long msgSizeInBytes) {
@Override
public void resetPublishCountAndEnableReadIfRequired() {
if (this.publishRateLimiter.resetPublishCount()) {
-enableProduerRead();
+enableProducerRead();
}
}
/**
* it sets cnx auto-readable if producer's cnx is disabled due to
publish-throttling
*/
-protected void enableProduerRead() {
+protected void enableProducerRead() {
if (producers != null) {
-producers.forEach(producer ->
producer.getCnx().enableCnxAutoRead());
+producers.values().forEach(producer ->
producer.getCnx().enableCnxAutoRead());
+}
+}
+
+protected void checkTopicFenced() throws BrokerServiceException {
+if (isFenced) {
+log.warn("[{}] Attempting to add producer to a fenced topic",
topic);
+throw new BrokerServiceException.TopicFencedException("Topic is
temporarily unavailable");
+}
+}
+
+protected void internalAddProducer(Producer producer) throws
BrokerServiceException {
+if (isProducersExceeded()) {
+log.warn("[{}] Attempting to add producer to topic which reached
max producers limit", topic);
+throw new BrokerServiceException.ProducerBusyException("Topic
reached max producers limit");
+}
+
+if (log.isDebugEnabled()) {
+log.debug("[{}] {} Got request to create producer ", topic,
producer.getProducerName());
+}
+
+Producer existProducer =
producers.putIfAbsent(producer.getProducerName(), producer);
+if (existProducer != null) {
+tryOverwriteOldProducer(existProducer, producer);
+}
+}
+
+private void tryOverwriteOldProducer(Producer oldProducer, Producer
newProducer)
+throws BrokerServiceException {
+boolean canOverwrite = false;
+if (oldProducer.equals(newProducer) &&
!oldProducer.isUserProvidedProducerName()
+&& !newProducer.isUserProvidedProducerName() &&
newProducer.getEpoch() > oldProducer.getEpoch()) {
+oldProducer.close();
+canOverwrite = true;
+}
+if (canOverwrite) {
Review comment:
May be if can simplified with
```
if (!canOverwrite || !producers.replace(newProducer.getProducerName(),
oldProducer, newProducer)) {
throw new BrokerServiceException.NamingException(
"Producer with name '" + newProducer.getProducerName() +
"' is already connected")
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] codelipenghui commented on a change in pull request #5571: Add epoch for connection handler to handle create producer timeout.
codelipenghui commented on a change in pull request #5571: Add epoch for
connection handler to handle create producer timeout.
URL: https://github.com/apache/pulsar/pull/5571#discussion_r344017172
##
File path:
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
##
@@ -258,16 +258,59 @@ public void incrementPublishCount(int numOfMessages,
long msgSizeInBytes) {
@Override
public void resetPublishCountAndEnableReadIfRequired() {
if (this.publishRateLimiter.resetPublishCount()) {
-enableProduerRead();
+enableProducerRead();
}
}
/**
* it sets cnx auto-readable if producer's cnx is disabled due to
publish-throttling
*/
-protected void enableProduerRead() {
+protected void enableProducerRead() {
if (producers != null) {
-producers.forEach(producer ->
producer.getCnx().enableCnxAutoRead());
+producers.values().forEach(producer ->
producer.getCnx().enableCnxAutoRead());
+}
+}
+
+protected void checkTopicFenced() throws BrokerServiceException {
+if (isFenced) {
+log.warn("[{}] Attempting to add producer to a fenced topic",
topic);
+throw new BrokerServiceException.TopicFencedException("Topic is
temporarily unavailable");
+}
+}
+
+protected void internalAddProducer(Producer producer) throws
BrokerServiceException {
+if (isProducersExceeded()) {
+log.warn("[{}] Attempting to add producer to topic which reached
max producers limit", topic);
+throw new BrokerServiceException.ProducerBusyException("Topic
reached max producers limit");
+}
+
+if (log.isDebugEnabled()) {
+log.debug("[{}] {} Got request to create producer ", topic,
producer.getProducerName());
+}
+
+Producer existProducer =
producers.putIfAbsent(producer.getProducerName(), producer);
+if (existProducer != null) {
+tryOverwriteOldProducer(existProducer, producer);
+}
+}
+
+private void tryOverwriteOldProducer(Producer oldProducer, Producer
newProducer)
+throws BrokerServiceException {
+boolean canOverwrite = false;
+if (oldProducer.equals(newProducer) &&
!oldProducer.isUserProvidedProducerName()
+&& !newProducer.isUserProvidedProducerName() &&
newProducer.getEpoch() > oldProducer.getEpoch()) {
+oldProducer.close();
+canOverwrite = true;
+}
+if (canOverwrite) {
Review comment:
May be if can simplified with
```
if (!canOverwrite || !producers.replace(newProducer.getProducerName(),
oldProducer, newProducer)) {
throw new BrokerServiceException.NamingException(
"Producer with name '" + newProducer.getProducerName() +
"' is already connected")
}
```
Right?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] Sunkwan-Kwon opened a new pull request #5592: Fixes #5589
Sunkwan-Kwon opened a new pull request #5592: Fixes #5589 URL: https://github.com/apache/pulsar/pull/5592 ### Motivation It seems that there is a memory leak in the pulsar-function-go library. I implemented a simple pulsar function worker that just write logs using pulsar-function-go/logutil for sending logs to log topic. I tried to long-term test by sending request messages consecutively to the input topic to check the feasibility. During the test, I faced `ProducerQueueIsFull` error with `--log-topic` option. And I observed indefinitely grown memory usage of the pulsar function worker process. Please refer to the issue for more detail. (#5589) ### Modifications Clear the `StrEntry` variable after finish addLogTopicHandler() function regardless of the log messages are appended to logger or not. If it is not cleared, it causes memory leak because StrEntry has grown indefinitely. Moreover, if the function set --log-topic, then the topic could get accumulated huge messages that cause ProducerQueueIsFull error. ### Verifying this change Verified it by reproducing step described in the issue #5589. ### Does this pull request potentially affect one of the following parts: *If `yes` was chosen, please highlight the changes* - Dependencies (does it add or upgrade a dependency): (no) - The public API: (no) - The schema: (no) - The default values of configurations: (no) - The wire protocol: (no) - The rest endpoints: (no) - The admin cli options: (no) - Anything that affects deployment: (no) ### Documentation - Does this pull request introduce a new feature? (no) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] krnaveen14 commented on a change in pull request #5591: KafkaSourceRecord - Initialize key property with Optional.empty()
krnaveen14 commented on a change in pull request #5591: KafkaSourceRecord -
Initialize key property with Optional.empty()
URL: https://github.com/apache/pulsar/pull/5591#discussion_r344016513
##
File path:
pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/KafkaConnectSource.java
##
@@ -187,9 +187,7 @@ public void close() {
srcRecord.topic(), srcRecord.keySchema(), srcRecord.key());
byte[] valueBytes = valueConverter.fromConnectData(
srcRecord.topic(), srcRecord.valueSchema(), srcRecord.value());
-if (keyBytes != null) {
-this.key =
Optional.of(Base64.getEncoder().encodeToString(keyBytes));
-}
+this.key = keyBytes != null ?
Optional.of(Base64.getEncoder().encodeToString(keyBytes)) : Optional.empty();
Review comment:
No it doesn't.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] sijie commented on issue #4621: [PIP-38] Support batch receive in java client.
sijie commented on issue #4621: [PIP-38] Support batch receive in java client. URL: https://github.com/apache/pulsar/pull/4621#issuecomment-551417215 @merlimat since you were involved in the review before, can you please take a look again? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] sijie commented on a change in pull request #5591: KafkaSourceRecord - Initialize key property with Optional.empty()
sijie commented on a change in pull request #5591: KafkaSourceRecord -
Initialize key property with Optional.empty()
URL: https://github.com/apache/pulsar/pull/5591#discussion_r344016034
##
File path:
pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/KafkaConnectSource.java
##
@@ -187,9 +187,7 @@ public void close() {
srcRecord.topic(), srcRecord.keySchema(), srcRecord.key());
byte[] valueBytes = valueConverter.fromConnectData(
srcRecord.topic(), srcRecord.valueSchema(), srcRecord.value());
-if (keyBytes != null) {
-this.key =
Optional.of(Base64.getEncoder().encodeToString(keyBytes));
-}
+this.key = keyBytes != null ?
Optional.of(Base64.getEncoder().encodeToString(keyBytes)) : Optional.empty();
Review comment:
does `encodeToString` accept `null`? If so, can we just use `ofNullable`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] sijie commented on issue #5590: [Issue 5474][pulsar-io-debezium] Support CDC Connector for MongoDB
sijie commented on issue #5590: [Issue 5474][pulsar-io-debezium] Support CDC Connector for MongoDB URL: https://github.com/apache/pulsar/pull/5590#issuecomment-551416066 @tuteng @murong00 PTAL This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] krnaveen14 opened a new pull request #5591: KafkaSourceRecord - Initialize key property with Optional.empty()
krnaveen14 opened a new pull request #5591: KafkaSourceRecord - Initialize key property with Optional.empty() URL: https://github.com/apache/pulsar/pull/5591 [pulsar-io] [kafka-connect-adapter] KafkaSourceRecord - Initialize key property with Optional.empty() if keyBytes from SourceRecord is null (Optional property should never itself be null) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] yuanjingshi commented on issue #5568: Function worker fails to be authenticated when TLS authentication is enabled in Pulsar standalone
yuanjingshi commented on issue #5568: Function worker fails to be authenticated when TLS authentication is enabled in Pulsar standalone URL: https://github.com/apache/pulsar/issues/5568#issuecomment-551415259 @jiazhai Set useTls to be true is only a work around since useTls is already depreciated with the intention to use url as a set up of enabling TLS authentication. The future versions shall honour this change and be inconsistent with the configuration. Using a depreciated config is rolling back. Since it was depreciated, necessary code changes are required to remove the dependency of it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] huangdx0726 opened a new pull request #5590: support mongodb connector
huangdx0726 opened a new pull request #5590: support mongodb connector URL: https://github.com/apache/pulsar/pull/5590 ## [Issue 5474][pulsar-io-debezium] Support CDC Connector for MongoDB Fixes #5474 ### Motivation Support CDC Connector for MongoDB ### Modifications add mongodb module in pular-io-debezium component . ### Does this pull request potentially affect one of the following parts: *If `yes` was chosen, please highlight the changes* - Dependencies (does it add or upgrade a dependency): (no) - The public API: (yes / no) - The schema: (yes / no / don't know) - The default values of configurations: (no) - The wire protocol: ( no) - The rest endpoints: ( no) - The admin cli options: ( no) - Anything that affects deployment: ( no) ### Documentation - Does this pull request introduce a new feature? ( no) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] sijie commented on issue #5571: Add epoch for connection handler to handle create producer timeout.
sijie commented on issue #5571: Add epoch for connection handler to handle create producer timeout. URL: https://github.com/apache/pulsar/pull/5571#issuecomment-551414551 @merlimat @rdhabalia PTAL This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] sijie commented on a change in pull request #5571: Add epoch for connection handler to handle create producer timeout.
sijie commented on a change in pull request #5571: Add epoch for connection
handler to handle create producer timeout.
URL: https://github.com/apache/pulsar/pull/5571#discussion_r344014053
##
File path:
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
##
@@ -258,16 +258,59 @@ public void incrementPublishCount(int numOfMessages,
long msgSizeInBytes) {
@Override
public void resetPublishCountAndEnableReadIfRequired() {
if (this.publishRateLimiter.resetPublishCount()) {
-enableProduerRead();
+enableProducerRead();
}
}
/**
* it sets cnx auto-readable if producer's cnx is disabled due to
publish-throttling
*/
-protected void enableProduerRead() {
+protected void enableProducerRead() {
if (producers != null) {
-producers.forEach(producer ->
producer.getCnx().enableCnxAutoRead());
+producers.values().forEach(producer ->
producer.getCnx().enableCnxAutoRead());
+}
+}
+
+protected void checkTopicFenced() throws BrokerServiceException {
+if (isFenced) {
+log.warn("[{}] Attempting to add producer to a fenced topic",
topic);
+throw new BrokerServiceException.TopicFencedException("Topic is
temporarily unavailable");
+}
+}
+
+protected void internalAddProducer(Producer producer) throws
BrokerServiceException {
+if (isProducersExceeded()) {
+log.warn("[{}] Attempting to add producer to topic which reached
max producers limit", topic);
+throw new BrokerServiceException.ProducerBusyException("Topic
reached max producers limit");
+}
+
+if (log.isDebugEnabled()) {
+log.debug("[{}] {} Got request to create producer ", topic,
producer.getProducerName());
+}
+
+Producer existProducer =
producers.putIfAbsent(producer.getProducerName(), producer);
+if (existProducer != null) {
+tryOverwriteOldProducer(existProducer, producer);
+}
+}
+
+private void tryOverwriteOldProducer(Producer oldProducer, Producer
newProducer)
+throws BrokerServiceException {
+boolean canOverwrite = false;
+if (oldProducer.equals(newProducer) &&
!oldProducer.isUserProvidedProducerName()
+&& !newProducer.isUserProvidedProducerName() &&
newProducer.getEpoch() > oldProducer.getEpoch()) {
+oldProducer.close();
+canOverwrite = true;
+}
+if (canOverwrite) {
Review comment:
nit: this can be simplified with
```
if (canOverwrite && !producers.replace(newProducer.getProducerName(),
oldProducer, newProducer)) {
throw new BrokerServiceException.NamingException(
"Producer with name '" + newProducer.getProducerName() +
"' is already connected
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[pulsar.wiki] branch master updated: Created PIP-51 Introduce sticky consumer (markdown)
This is an automated email from the ASF dual-hosted git repository. penghui pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.wiki.git The following commit(s) were added to refs/heads/master by this push: new 4541651 Created PIP-51 Introduce sticky consumer (markdown) 4541651 is described below commit 45416510b4353a92e25e6ad1046bfdfb3e099705 Author: lipenghui AuthorDate: Fri Nov 8 15:04:54 2019 +0800 Created PIP-51 Introduce sticky consumer (markdown) --- PIP-51-Introduce-sticky-consumer.md | 89 + 1 file changed, 89 insertions(+) diff --git a/PIP-51-Introduce-sticky-consumer.md b/PIP-51-Introduce-sticky-consumer.md new file mode 100644 index 000..7586be2 --- /dev/null +++ b/PIP-51-Introduce-sticky-consumer.md @@ -0,0 +1,89 @@ +- Status: Draft +- Author: Penghui Li +- Discussion Thread: +- Pull Request: + +## Motivation +In Key_shared subscription, there can be more consumers than partitions and the messages of the same key are routed to one consumer of the subscription in order. + +Currently, changes of membership in Key_shared subscription will result in hash range reassign of active consumers, it means changes of membership lead to changes of key's owner consumer. + +This proposal aims to introduce sticky consumer based on Key_shared subscription. Sticky consumer allows more consumers than partitions and the same key message distribution order guarantee. The difference between sticky consumer is a consumer will be responsible for fixed keys and this can be specified by the user. Changes of membership in subscription does not cause the key's consumer change. + +Many scenarios can take advantage of this feature, here are a few examples detailing how to take advantage of this feature. + +**Consumer grayscale release** + +Enterprise applications are constantly being updated to optimize the business or user experience, and consumers are constantly optimising. Enterprises often need to verify the effect of optimization through grayscale publishing, and the consumer can use this feature to get convenience. +With sticky consumer: +Create a Key_shared subscription hash range size 100. +Old consumer consume messages from slots [0 - 98] . +New consumer consume messages from slot [99 - 99]. +Without sticky consumer: +Create 2 subscriptions with different subscription name and consumer implements its own filter. +Compare these two ways, with sticky consumer avoiding wasted resources for repeated message delivery and bringing convenient tools to developers. + +## Approach +Introducing exclusive key hash range mechanism to approach sticky consumer. While a new subscription create, a new fixed size key hash range container will create. + + + +There are three areas in the illustration, the yellow area represents consumer-1 responsible for the consumption of messages which routing to this range, the blue area represents consumer-2 responsible for the consumption of messages which routing to this range and the grey area represents this range not yet assigned. + +The routing of the message is determined according to the key of the message, is obtained by modulo operation of the total size of the hash ranges and hash of the key. The message has two keys: partition key and ordering key, ordering key has higher priority on message routing than partition key. But ordering key is not required, if no ordering key is specified, it will be routed according to the partition key. + +While a new consumer attempt to connect, the following situation will reject consumer: +1. All hash range is assigned to consumers +2. New consumer without responsible hash range +3. The responsible range of new consumer overlaps with the range that the existing consumer is responsible for. +4. New consumer with an invalid slot + +While consumer successfully establishes a connection and assign hash range, broker start dispatch messages to the consumer in order. + +While a consumer disconnected, hash range assigned to this consumer will change to unassigned state. + + + +As shown in the illustration, consumer-1 disconnected from broker and the range consumer-1 responsible for changed to unassigned state. The Broker will no longer continue to dispatch messages to this hash range until a new consumer is added and the hash range assigned to it. + +Here is a simple usecase : +User create 3 consumers to subscribe a topic with Key_shared subscription and configured with sticky consumer, the subscription with 10 hash range size and consumer-1 consume messages from hash range [0 - 3], consumer-2 consume messages from hash range [4 - 5] and consumer-3 consume messages from hash range [6 - 9]. + +
Showing nodes accounting for 12.51MB, 100% of 12.51MB total
flat flat% sum%cum cum%
12.51MB 100% 100%12.51MB 100%
github.com/sirupsen/logrus.(*Entry).String
0 0% 100%12.51MB 100%
github.com/apache/pulsar/pulsar-function-go/logutil.(*contextHook).Fire
0 0% 100%1MB 7.99%
github.com/apache/pulsar/pulsar-function-go/logutil.Error
0 0% 100%11.51MB 92.01%
github.com/apache/pulsar/pulsar-function-go/logutil.Infof
0 0% 100%1MB 7.99%
github.com/apache/pulsar/pulsar-function-go/pf.(*goInstance).addLogTopicHandler
0 0% 100%11.51MB 92.01%
github.com/apache/pulsar/pulsar-function-go/pf.(*goInstance).handlerMsg
0 0% 100%12.51MB 100%
github.com/apache/pulsar/pulsar-function-go/pf.(*goInstance).startFunction
0 0% 100%12.51MB 100%
github.com/apache/pulsar/pulsar-function-go/pf.Start
0 0% 100%11.51MB 92.01%
github.com/apache/pulsar/pulsar-function-go/pf.newFunction.func1
0 0% 100%11.51MB 92.01%
github.com/apache/pulsar/pulsar-function-go/pf.pulsarFunction.process
0 0% 100%12.51MB 100%
github.com/sirupsen/logrus.(*Entry).Log
0 0% 100%11.51MB 92.01%
github.com/sirupsen/logrus.(*Entry).Logf
...
...
```
As you could see, a string object for logging had grown during the test.
It seems that `ProducerQueueIsFull` error with `--log-topic` option, and the
memory leak without `--log-topic` option is caused by the same reason. So I
modified `pulsar-go-function` codes to fix it. Fortunately, It seems has been
fixed now. So I will send a pull request for that.
**To Reproduce**
Steps to reproduce the behavior:
1. Prepare a pulsar cluster with standalone mode.
2. Prepare a simple pulsar function worker. Refer to the codes below.
```
$ cat simple-worker.go
package main
import (
"context"
"fmt"
"net/http"
_ "net/http/pprof"
log "github.com/apache/pulsar/pulsar-function-go/logutil"
"github.com/apache/pulsar/pulsar-function-go/pf"
)
func main() {
// go routine for pprof
go func() {
fmt.Println("%+v", http.ListenAndServe("localhost:6060",
nil))
}()
pf.Start(testFunction)
}
func testFunction(ctx context.Context) {
for i := 0; i < 10; i++ {
log.Infof("This is test of pulsar function. This is test of
pulsar function. This is test of pulsar function. This is test of pulsar
function. This is test of pulsar function. This is test of pulsar function.
This is test of pulsar function. This is test of pulsar function. This is test
of pulsar function. This is test of pulsar function. This is test of pulsar
function. This is test of pulsar function. This is test of pulsar function.
This is test of pulsar function. This is test of pulsar function. This is test
of pulsar function. This is test of pulsar function. This is test of pulsar
function. This is test of pulsar function. This is test of pulsar function.
This is test of pulsar function. This is test of pulsar function. This is test
of pulsar function. This is test of pulsar function. This is test of pulsar
function. This is test of pulsar function. This is test of pulsar function.
This is test of pulsar function. This is test of pulsar function.\n")
}
}
```
3. Prepare a conf.yaml file for the function worker.
```
---
pulsarServiceURL: "pulsar://sunkwan-devpc:6650"
instanceID: 0
[GitHub] [pulsar-translation] Jennifer88huang merged pull request #7: update schedule
Jennifer88huang merged pull request #7: update schedule URL: https://github.com/apache/pulsar-translation/pull/7 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[pulsar-translation] branch master updated: update schedule (#7)
This is an automated email from the ASF dual-hosted git repository. hjf pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar-translation.git The following commit(s) were added to refs/heads/master by this push: new 3d0bd59 update schedule (#7) 3d0bd59 is described below commit 3d0bd5948355ead678f6701b9681ef8003bc33c9 Author: Sylvia <39793568+sylviab...@users.noreply.github.com> AuthorDate: Fri Nov 8 14:48:38 2019 +0800 update schedule (#7) * update * Update schedule.md * Update schedule.md * Update schedule.md --- schedule.md | 48 1 file changed, 48 insertions(+) diff --git a/schedule.md b/schedule.md index c6b7035..c1074a2 100644 --- a/schedule.md +++ b/schedule.md @@ -41,3 +41,51 @@ io-redis | Bu Xing | | In Progress io-solr | Bu Xing | | In Progress io-tcp | Bu Xing | | In Progress security-extending | bilahepan(Tianci Gao) | | In Progress +getting-started-standalone (Correspond 'Run Pulsar locally') | | | To Do +getting-started-docker | | | To Do +getting-started-clients | | | To Do +schema-get-started | | | To Do +schema-understand | | | To Do +schema-evolution-compatibility | | | To Do +schema-manage | | | To Do +functions-overview | | | To Do +functions-worker | | | To Do +functions-runtime | | | To Do +functions-develop | | | To Do +functions-debug | | | To Do +functions-deploy | | | To Do +functions-cli | | | To Do +io-overview | | | To Do +io-quickstart (Correspond 'Get started') | | | To Do +io-use | | | To Do +io-debug | | | To Do +io-connectors | SylviaBABY | | In Progress +io-cdc | SylviaBABY | | In Progress +io-develop | | | To Do +io-cli | | | To Do +io-aerospike-sink | | | To Do +io-canal-source | | | To Do +io-cassandra-sink | | | To Do +io-cdc-canal | | | To Do +io-cdc-debezium | | | To Do +io-debezium-source | | | To Do +io-elasticsearch-sink | | | To Do +io-file-source | | | To Do +io-flume-sink | | | To Do +io-flume-source | | | To Do +io-hbase | | | To Do +io-hdfs2-sink | | | To Do +io-hdfs3-sink | | | To Do +io-influxdb-sink | | | To Do +io-jdbc-sink | | | To Do +io-kafka-sink | | | To Do +io-kafka-source | | | To Do +io-kinesis-sink | | | To Do +io-kinesis-source | | | To Do +io-mongo-sink | | | To Do +io-netty-source | | | To Do +io-rabbitmq-sink | | | To Do +io-rabbitmq-source | | | To Do +io-redis-sink | | | To Do +io-solr-sink | | | To Do +io-twitter-source | | | To Do
[GitHub] [pulsar] huangdx0726 commented on issue #5474: Support CDC Connector for MongoDB
huangdx0726 commented on issue #5474: Support CDC Connector for MongoDB URL: https://github.com/apache/pulsar/issues/5474#issuecomment-551403069 ok ，i got it This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[pulsar] branch master updated (c460f22 -> 951664c)
This is an automated email from the ASF dual-hosted git repository.
sijie pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git.
from c460f22 Remove old Pulsar website (#5576)
add 951664c [PIP-51] Introduce sticky consumer (#5388)
No new revisions were added by this update.
Summary of changes:
.../org/apache/pulsar/broker/service/Consumer.java | 11 +-
...shRangeAutoSplitStickyKeyConsumerSelector.java} |8 +-
...ashRangeExclusiveStickyKeyConsumerSelector.java | 118 +
.../apache/pulsar/broker/service/ServerCnx.java|6 +-
.../broker/service/StickyKeyConsumerSelector.java |9 +
.../org/apache/pulsar/broker/service/Topic.java|3 +-
...istentStickyKeyDispatcherMultipleConsumers.java | 12 +-
.../nonpersistent/NonPersistentSubscription.java | 22 +-
.../service/nonpersistent/NonPersistentTopic.java |5 +-
...istentStickyKeyDispatcherMultipleConsumers.java | 12 +-
.../service/persistent/PersistentSubscription.java | 22 +-
.../broker/service/persistent/PersistentTopic.java |5 +-
...ngeAutoSplitStickyKeyConsumerSelectorTest.java} | 12 +-
...angeExclusiveStickyKeyConsumerSelectorTest.java | 191 +
.../PersistentDispatcherFailoverConsumerTest.java | 18 +-
.../service/PersistentTopicConcurrentTest.java |8 +-
.../pulsar/broker/service/PersistentTopicTest.java | 61 +-
.../client/api/KeySharedSubscriptionTest.java | 181 +-
.../NonPersistentKeySharedSubscriptionTest.java| 181 +-
.../apache/pulsar/client/api/ConsumerBuilder.java | 27 +-
...CryptoFailureAction.java => KeySharedMode.java} | 15 +-
.../apache/pulsar/client/api/KeySharedPolicy.java | 112 +
.../java/org/apache/pulsar/client/api/Range.java | 54 +-
.../pulsar/client/impl/ConsumerBuilderImpl.java| 14 +
.../apache/pulsar/client/impl/ConsumerImpl.java|2 +-
.../impl/conf/ConsumerConfigurationData.java |3 +
.../pulsar/client/api/KeySharedPolicyTest.java | 62 +
.../org/apache/pulsar/client/api/RangeTest.java| 64 +
.../apache/pulsar/common/api/proto/PulsarApi.java | 7948 +++-
.../apache/pulsar/common/protocol/Commands.java| 33 +
pulsar-common/src/main/proto/PulsarApi.proto | 21 +-
31 files changed, 5627 insertions(+), 3613 deletions(-)
rename
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/{HashRangeStickyKeyConsumerSelector.java
=> HashRangeAutoSplitStickyKeyConsumerSelector.java} (95%)
create mode 100644
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/HashRangeExclusiveStickyKeyConsumerSelector.java
rename
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/{HashRangeStickyKeyConsumerSelectorTest.java
=> HashRangeAutoSplitStickyKeyConsumerSelectorTest.java} (92%)
create mode 100644
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/HashRangeExclusiveStickyKeyConsumerSelectorTest.java
copy
pulsar-client-api/src/main/java/org/apache/pulsar/client/api/{ProducerCryptoFailureAction.java
=> KeySharedMode.java} (74%)
create mode 100644
pulsar-client-api/src/main/java/org/apache/pulsar/client/api/KeySharedPolicy.java
copy
pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarConnectorId.java
=> pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Range.java
(50%)
create mode 100644
pulsar-client/src/test/java/org/apache/pulsar/client/api/KeySharedPolicyTest.java
create mode 100644
pulsar-client/src/test/java/org/apache/pulsar/client/api/RangeTest.java
[GitHub] [pulsar] sijie merged pull request #5388: [PIP-51] Introduce sticky consumer
sijie merged pull request #5388: [PIP-51] Introduce sticky consumer URL: https://github.com/apache/pulsar/pull/5388 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] Jennifer88huang commented on issue #5474: Support CDC Connector for MongoDB
Jennifer88huang commented on issue #5474: Support CDC Connector for MongoDB URL: https://github.com/apache/pulsar/issues/5474#issuecomment-551399484 @huangdx0726 Sure, thank you very much for your contribution. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] codelipenghui commented on issue #5388: [PIP-51] Introduce sticky consumer
codelipenghui commented on issue #5388: [PIP-51] Introduce sticky consumer URL: https://github.com/apache/pulsar/pull/5388#issuecomment-551398978 @sijie I have updated the enum in PulsarApi.proto This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] huangdx0726 commented on issue #5474: Support CDC Connector for MongoDB
huangdx0726 commented on issue #5474: Support CDC Connector for MongoDB URL: https://github.com/apache/pulsar/issues/5474#issuecomment-551398948 @tuteng May i try it？ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jiazhai commented on issue #5568: Function worker fails to be authenticated when TLS authentication is enabled in Pulsar standalone
jiazhai commented on issue #5568: Function worker fails to be authenticated when TLS authentication is enabled in Pulsar standalone URL: https://github.com/apache/pulsar/issues/5568#issuecomment-551391440 @yuanjingshi Thanks for reporting this issue. this should be a documentation issue. Once broker is set to use TLS, function worker should also set useTls to true. In function worker, there are some internal pulsar-client and pulsar-manager which need to auth with broker. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] Jennifer88huang closed issue #5101: Add documentation of pulsar-client-node to website
Jennifer88huang closed issue #5101: Add documentation of pulsar-client-node to website URL: https://github.com/apache/pulsar/issues/5101 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] Jennifer88huang commented on issue #5101: Add documentation of pulsar-client-node to website
Jennifer88huang commented on issue #5101: Add documentation of pulsar-client-node to website URL: https://github.com/apache/pulsar/issues/5101#issuecomment-551391128 Fixed in #5212 , so close this issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] sijie commented on a change in pull request #5574: Add Github workflow for gated checkin
sijie commented on a change in pull request #5574: Add Github workflow for gated checkin URL: https://github.com/apache/pulsar/pull/5574#discussion_r343993316 ## File path: .github/workflows/cpp-ci.yaml ## @@ -0,0 +1,44 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +name: CI +on: [pull_request] + +jobs: + + cpp-integration: +name: C++ / Python tests +runs-on: ubuntu-latest + +steps: + - name: Set up JDK 1.8 +uses: actions/setup-java@v1 +with: + java-version: 1.8 + + - name: Checkout code +uses: actions/checkout@master Review comment: ``` By default, this is equivalent to running git fetch and git checkout $GITHUB_SHA, so that you'll always have your repo contents at the version that triggered the workflow. ``` It doesn't work exactly same as the github plugin. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jiazhai commented on issue #5558: When we use the pulsar-admin.sh script to delete partitioned topic, it will throw the http error, error code is 500
jiazhai commented on issue #5558: When we use the pulsar-admin.sh script to delete partitioned topic, it will throw the http error, error code is 500 URL: https://github.com/apache/pulsar/issues/5558#issuecomment-551379679 need handling the exception reported This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] codelipenghui commented on issue #5388: [PIP-51] Introduce sticky consumer
codelipenghui commented on issue #5388: [PIP-51] Introduce sticky consumer URL: https://github.com/apache/pulsar/pull/5388#issuecomment-551372407 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] congbobo184 commented on issue #5570: Transaction log implemention
congbobo184 commented on issue #5570: Transaction log implemention URL: https://github.com/apache/pulsar/pull/5570#issuecomment-551369493 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jiazhai commented on issue #3833: Error offloading:
jiazhai commented on issue #3833: Error offloading: URL: https://github.com/apache/pulsar/issues/3833#issuecomment-551368030 will check and handle this issue This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jiazhai commented on issue #5585: Non-persistent topic's replication has a deadlock
jiazhai commented on issue #5585: Non-persistent topic's replication has a deadlock URL: https://github.com/apache/pulsar/issues/5585#issuecomment-551366825 👍 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar-translation] SylviaBABY commented on issue #7: update schedule
SylviaBABY commented on issue #7: update schedule URL: https://github.com/apache/pulsar-translation/pull/7#issuecomment-551366698 @Anonymitaet update,pls check This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jiazhai commented on issue #5582: pulsar dashboard collector encountered error message
jiazhai commented on issue #5582: pulsar dashboard collector encountered error message URL: https://github.com/apache/pulsar/issues/5582#issuecomment-551364856 @tuteng would you please help take a look of this issue? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] Jennifer88huang commented on issue #5327: typo: traffic
Jennifer88huang commented on issue #5327: typo: traffic URL: https://github.com/apache/pulsar/pull/5327#issuecomment-551364431 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] codelipenghui commented on a change in pull request #5587: [Issue 5585][pulsar-client] Fix producer Semaphore release error
codelipenghui commented on a change in pull request #5587: [Issue
5585][pulsar-client] Fix producer Semaphore release error
URL: https://github.com/apache/pulsar/pull/5587#discussion_r343967369
##
File path:
pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
##
@@ -795,7 +795,7 @@ void ackReceived(ClientCnx cnx, long sequenceId, long
ledgerId, long entryId) {
log.debug("[{}] [{}] Received ack for msg {} ", topic,
producerName, sequenceId);
}
pendingMessages.remove();
-semaphore.release(op.numMessagesInBatch);
+semaphore.release(isBatchMessagingEnabled() ?
op.numMessagesInBatch : 1);
Review comment:
The numMessagesInBatch also used by client stats calculation, for example,
if the metadata of message with numMessagesInBatch=10 then we set the
numMessagesInBatch=1 of op, the stats of this producer will be wrong.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] weishuisheng commented on a change in pull request #5587: [Issue 5585][pulsar-client] Fix producer Semaphore release error
weishuisheng commented on a change in pull request #5587: [Issue
5585][pulsar-client] Fix producer Semaphore release error
URL: https://github.com/apache/pulsar/pull/5587#discussion_r343967222
##
File path:
pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
##
@@ -1280,12 +1279,15 @@ private void failPendingMessages(ClientCnx cnx,
PulsarClientException ex) {
ReferenceCountUtil.safeRelease(op.cmd);
op.recycle();
});
-semaphore.release(releaseCount.get());
+
pendingMessages.clear();
pendingCallbacks.clear();
if (isBatchMessagingEnabled()) {
failPendingBatchMessages(ex);
}
+
+semaphore.drainPermits();
+semaphore.release(conf.getMaxPendingMessages());
Review comment:
You are right, I have fixed it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] cdbartholomew commented on issue #5360: Proxy doesn't send request body on redirects
cdbartholomew commented on issue #5360: Proxy doesn't send request body on redirects URL: https://github.com/apache/pulsar/issues/5360#issuecomment-551352913 This is definitely what is happening in #3702. I was just debugging that with tcpdump and ended up here. Great to see a fix is on its way. Thanks @addisonj and @merlimat. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] weishuisheng commented on a change in pull request #5587: [Issue 5585][pulsar-client] Fix producer Semaphore release error
weishuisheng commented on a change in pull request #5587: [Issue
5585][pulsar-client] Fix producer Semaphore release error
URL: https://github.com/apache/pulsar/pull/5587#discussion_r343961280
##
File path:
pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
##
@@ -1280,12 +1279,15 @@ private void failPendingMessages(ClientCnx cnx,
PulsarClientException ex) {
ReferenceCountUtil.safeRelease(op.cmd);
op.recycle();
});
-semaphore.release(releaseCount.get());
+
pendingMessages.clear();
pendingCallbacks.clear();
if (isBatchMessagingEnabled()) {
failPendingBatchMessages(ex);
}
+
+semaphore.drainPermits();
+semaphore.release(conf.getMaxPendingMessages());
Review comment:
OK,I will fix it
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] weishuisheng commented on a change in pull request #5587: [Issue 5585][pulsar-client] Fix producer Semaphore release error
weishuisheng commented on a change in pull request #5587: [Issue
5585][pulsar-client] Fix producer Semaphore release error
URL: https://github.com/apache/pulsar/pull/5587#discussion_r343961195
##
File path:
pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
##
@@ -952,6 +952,7 @@ void recycle() {
rePopulate = null;
sequenceId = -1;
createdAt = -1;
+numMessagesInBatch = 1;
Review comment:
Got it,I will remove it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] merlimat commented on a change in pull request #5587: [Issue 5585][pulsar-client] Fix producer Semaphore release error
merlimat commented on a change in pull request #5587: [Issue
5585][pulsar-client] Fix producer Semaphore release error
URL: https://github.com/apache/pulsar/pull/5587#discussion_r343932673
##
File path:
pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
##
@@ -795,7 +795,7 @@ void ackReceived(ClientCnx cnx, long sequenceId, long
ledgerId, long entryId) {
log.debug("[{}] [{}] Received ack for msg {} ", topic,
producerName, sequenceId);
}
pendingMessages.remove();
-semaphore.release(op.numMessagesInBatch);
+semaphore.release(isBatchMessagingEnabled() ?
op.numMessagesInBatch : 1);
Review comment:
or, can we just make sure for `op.numMessagesInBatch` to be set correctly
in both cases?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] merlimat commented on a change in pull request #5587: [Issue 5585][pulsar-client] Fix producer Semaphore release error
merlimat commented on a change in pull request #5587: [Issue
5585][pulsar-client] Fix producer Semaphore release error
URL: https://github.com/apache/pulsar/pull/5587#discussion_r343932980
##
File path:
pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
##
@@ -952,6 +952,7 @@ void recycle() {
rePopulate = null;
sequenceId = -1;
createdAt = -1;
+numMessagesInBatch = 1;
Review comment:
Rather than setting this here, we should set it correctly in the two
`create()` methods
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] merlimat commented on a change in pull request #5587: [Issue 5585][pulsar-client] Fix producer Semaphore release error
merlimat commented on a change in pull request #5587: [Issue
5585][pulsar-client] Fix producer Semaphore release error
URL: https://github.com/apache/pulsar/pull/5587#discussion_r343933356
##
File path:
pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
##
@@ -1280,12 +1279,15 @@ private void failPendingMessages(ClientCnx cnx,
PulsarClientException ex) {
ReferenceCountUtil.safeRelease(op.cmd);
op.recycle();
});
-semaphore.release(releaseCount.get());
+
pendingMessages.clear();
pendingCallbacks.clear();
if (isBatchMessagingEnabled()) {
failPendingBatchMessages(ex);
}
+
+semaphore.drainPermits();
+semaphore.release(conf.getMaxPendingMessages());
Review comment:
This doesn't look correct to me, what's the purpose of release 1000 permits
on the semaphore?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] merlimat commented on a change in pull request #5574: Add Github workflow for gated checkin
merlimat commented on a change in pull request #5574: Add Github workflow for
gated checkin
URL: https://github.com/apache/pulsar/pull/5574#discussion_r343852759
##
File path:
pulsar-broker/src/test/java/org/apache/pulsar/websocket/proxy/ProxyPublishConsumeTlsTest.java
##
@@ -76,6 +76,8 @@ public void setup() throws Exception {
config.setBrokerClientAuthenticationParameters("tlsCertFile:" +
TLS_CLIENT_CERT_FILE_PATH + ",tlsKeyFile:" + TLS_CLIENT_KEY_FILE_PATH);
config.setBrokerClientAuthenticationPlugin(AuthenticationTls.class.getName());
String lookupUrl = new URI("pulsar://localhost:" +
BROKER_PORT_TLS).toString();
+// Add the required number of thread for testing in low core count
machines
+config.setNumHttpServerThreads(6);
Review comment:
Since this will be a problem in any case, we should make sure we do
`Math.max(6, config.getNumHttpServerThreads())` instead of just setting it for
tests
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] merlimat commented on a change in pull request #5574: Add Github workflow for gated checkin
merlimat commented on a change in pull request #5574: Add Github workflow for gated checkin URL: https://github.com/apache/pulsar/pull/5574#discussion_r343852001 ## File path: .github/workflows/cpp-ci.yaml ## @@ -0,0 +1,44 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +name: CI Review comment: In the Github UI this will appear with only `CI` when collapsed:  Maybe we could use C++ in the outer name as well. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat commented on a change in pull request #5574: Add Github workflow for gated checkin
merlimat commented on a change in pull request #5574: Add Github workflow for gated checkin URL: https://github.com/apache/pulsar/pull/5574#discussion_r343852036 ## File path: .github/workflows/cpp-ci.yaml ## @@ -0,0 +1,44 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +name: CI +on: [pull_request] + +jobs: + + cpp-integration: +name: C++ / Python tests +runs-on: ubuntu-latest + +steps: + - name: Set up JDK 1.8 +uses: actions/setup-java@v1 +with: + java-version: 1.8 + + - name: Checkout code +uses: actions/checkout@master Review comment: does this fetches master or the PR's head? And does it merge it with master like in Jenkins case? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] sijie commented on issue #5574: Add Github workflow for gated checkin
sijie commented on issue #5574: Add Github workflow for gated checkin URL: https://github.com/apache/pulsar/pull/5574#issuecomment-551295320 > So, I would recommend to add more description on each PR so, it helps others to understand context and will be easy to review the PR. Fair enough. I will improve my PR in the future. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] aahmed-se commented on issue #5514: use pulsar get error in mac os
aahmed-se commented on issue #5514: use pulsar get error in mac os URL: https://github.com/apache/pulsar/issues/5514#issuecomment-551243188 @nathan-zhu try this ```pip3 install pulsar-client==2.4.1.post1``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] rdhabalia commented on issue #5574: Add Github workflow for gated checkin
rdhabalia commented on issue #5574: Add Github workflow for gated checkin URL: https://github.com/apache/pulsar/pull/5574#issuecomment-551205283 >> Since it was a fork of Ali ‘s pull quest #5064 to verify if GitHub actions works at a forked repo Sure, I was able to get the context from the previous PR. However, I just wanted to point out that many PRs in past from different committers don't have description and hard to get context from them for the review. Only specific group of committers have knowledge about it so, those PRs get approved and merged. So, I would recommend to add more description on each PR so, it helps others to understand context and will be easy to review the PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[pulsar] branch master updated (45bb182 -> c460f22)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 45bb182 Fix license file in the distribution package (#5578) add c460f22 Remove old Pulsar website (#5576) No new revisions were added by this update. Summary of changes: README.md |2 +- deployment/kubernetes/README.md|2 +- deployment/kubernetes/aws/README.md|2 +- distribution/server/src/assemble/src.xml |1 - pom.xml| 13 - site/.gitignore| 13 - site/.htmltest.yml | 26 - site/Gemfile | 23 - site/Gemfile.lock | 66 - site/Makefile | 100 - site/README.md | 179 -- site/VERSIONS |1 - site/_config.local.yml | 21 - site/_config.yml | 65 - site/_data/cli/bookkeeper.yaml | 73 - site/_data/cli/pulsar-admin.yaml | 909 --- site/_data/cli/pulsar-client.yaml | 63 - site/_data/cli/pulsar-daemon.yaml | 34 - site/_data/cli/pulsar-perf.yaml| 153 -- site/_data/cli/pulsar.yaml | 193 -- site/_data/codebase-ja.yaml| 58 - site/_data/codebase.yaml | 58 - site/_data/config/bookkeeper.yaml | 230 -- site/_data/config/broker.yaml | 355 --- site/_data/config/client.yaml | 35 - site/_data/config/discovery.yaml | 65 - site/_data/config/global_zookeeper.yaml| 44 - site/_data/config/log4j-shell.yaml | 39 - site/_data/config/log4j.yaml | 57 - site/_data/config/proxy.yaml | 81 - site/_data/config/standalone.yaml | 219 -- site/_data/config/websocket.yaml | 62 - site/_data/config/zookeeper.yaml | 44 - site/_data/connectors.yaml | 49 - site/_data/deps.yaml | 24 - site/_data/features.yaml | 60 - site/_data/messages-ja.yaml| 63 - site/_data/messages.yaml | 60 - site/_data/popovers-ja.yaml| 116 - site/_data/popovers.yaml | 124 - site/_data/resources/articles.yaml | 35 - site/_data/resources/presentations.yaml| 39 - site/_data/sidebar-ja.yaml | 113 - site/_data/sidebar.yaml| 203 -- site/_data/stats-ja.yaml | 69 - site/_data/stats.yaml | 70 - site/_includes/admonition.html | 26 - site/_includes/calendar.html | 22 - site/_includes/cli.html| 204 -- site/_includes/codebase-ja.html| 34 - site/_includes/codebase.html | 34 - site/_includes/config.html | 40 - site/_includes/connectors.html | 44 - site/_includes/explanations/admin-setup.md | 53 - site/_includes/explanations/client-url.md | 40 - site/_includes/explanations/deploying-bk.md| 59 - site/_includes/explanations/deploying-zk.md| 141 -- site/_includes/explanations/install-package.md | 69 - site/_includes/explanations/ja/admin-setup.md | 53 - site/_includes/explanations/ja/broker-admin.md | 170 -- site/_includes/explanations/ja/client-url.md | 54 - site/_includes/explanations/ja/cluster-admin.md| 205 -- site/_includes/explanations/ja/deploying-bk.md | 58 - site/_includes/explanations/ja/deploying-zk.md | 136 - site/_includes/explanations/ja/install-package.md | 75 - site/_includes/explanations/ja/instance-admin.md | 22 - site/_includes/explanations/ja/namespace-admin.md | 625 - .../explanations/ja/non-persistent-topic-admin.md | 239 -- .../explanations/ja/non-persistent-topics.md | 79 - .../explanations/ja/partitioned-topic-admin.md | 293 --- .../explanations/ja/partitioned-topics.md | 52 - site/_includes/explanations/ja/permissions.md | 135 - .../explanations/ja/persistent-topic-admin.md | 565 - .../explanations/ja/properties-namespaces.md | 42 - site/_includes/explanations/ja/property-admin.md | 98 - .../explanations/ja/service-d
[GitHub] [pulsar] merlimat merged pull request #5576: Remove old Pulsar website
merlimat merged pull request #5576: Remove old Pulsar website URL: https://github.com/apache/pulsar/pull/5576 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] codelipenghui commented on issue #5388: [PIP-51] Introduce sticky consumer
codelipenghui commented on issue #5388: [PIP-51] Introduce sticky consumer URL: https://github.com/apache/pulsar/pull/5388#issuecomment-551134668 @sijie I can update it tomorrow, my mac book has some problems with docker environment, can not build the proto. need to build them on another machine 😊 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] sijie commented on issue #5358: PIP-45: Switch ManagedLedger to use MetadataStore interface
sijie commented on issue #5358: PIP-45: Switch ManagedLedger to use MetadataStore interface URL: https://github.com/apache/pulsar/pull/5358#issuecomment-551132783 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] sijie commented on issue #5576: Remove old Pulsar website
sijie commented on issue #5576: Remove old Pulsar website URL: https://github.com/apache/pulsar/pull/5576#issuecomment-551128014 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] sijie commented on issue #5388: [PIP-51] Introduce sticky consumer
sijie commented on issue #5388: [PIP-51] Introduce sticky consumer URL: https://github.com/apache/pulsar/pull/5388#issuecomment-551127751 @codelipenghui can you update the enum? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] sijie commented on a change in pull request #5571: Add epoch for connection handler to handle create producer timeout.
sijie commented on a change in pull request #5571: Add epoch for connection
handler to handle create producer timeout.
URL: https://github.com/apache/pulsar/pull/5571#discussion_r343711137
##
File path:
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopic.java
##
@@ -207,9 +207,20 @@ public void addProducer(Producer producer) throws
BrokerServiceException {
log.debug("[{}] {} Got request to create producer ", topic,
producer.getProducerName());
}
-if (!producers.add(producer)) {
-throw new NamingException(
-"Producer with name '" + producer.getProducerName() +
"' is already connected to topic");
+Producer existProducer =
producers.putIfAbsent(producer.getProducerName(), producer);
+if (existProducer != null) {
+boolean canOverwrite = false;
+if (existProducer.equals(producer) &&
!existProducer.isUserProvidedProducerName()
+&& !producer.isUserProvidedProducerName() &&
producer.getEpoch() > existProducer.getEpoch()) {
+existProducer.close();
+canOverwrite = true;
+}
+if (canOverwrite) {
+producers.put(producer.getProducerName(), producer);
Review comment:
use `producers.replace(producer.getProducerName(), existingProducer,
producer)` to make sure one can successfully add the producer.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] sijie commented on a change in pull request #5571: Add epoch for connection handler to handle create producer timeout.
sijie commented on a change in pull request #5571: Add epoch for connection
handler to handle create producer timeout.
URL: https://github.com/apache/pulsar/pull/5571#discussion_r343711473
##
File path:
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
##
@@ -424,9 +424,20 @@ public void addProducer(Producer producer) throws
BrokerServiceException {
log.debug("[{}] {} Got request to create producer ", topic,
producer.getProducerName());
}
-if (!producers.add(producer)) {
-throw new NamingException(
-"Producer with name '" + producer.getProducerName() +
"' is already connected to topic");
+Producer existProducer =
producers.putIfAbsent(producer.getProducerName(), producer);
+if (existProducer != null) {
+boolean canOverwrite = false;
+if (existProducer.equals(producer) &&
!existProducer.isUserProvidedProducerName()
+&& !producer.isUserProvidedProducerName() &&
producer.getEpoch() > existProducer.getEpoch()) {
+existProducer.close();
+canOverwrite = true;
+}
+if (canOverwrite) {
+producers.put(producer.getProducerName(), producer);
Review comment:
use replace and check the return result
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] codelipenghui commented on issue #5587: [Issue 5585][pulsar-client] Fix producer Semaphore release error
codelipenghui commented on issue #5587: [Issue 5585][pulsar-client] Fix producer Semaphore release error URL: https://github.com/apache/pulsar/pull/5587#issuecomment-551125262 @wolfstudy if this PR can complete before cut 2.4.2, please considering include it, thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] codelipenghui commented on a change in pull request #5587: [Issue 5585][pulsar-client] Fix producer Semaphore release error
codelipenghui commented on a change in pull request #5587: [Issue
5585][pulsar-client] Fix producer Semaphore release error
URL: https://github.com/apache/pulsar/pull/5587#discussion_r343701349
##
File path:
pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
##
@@ -1280,12 +1279,15 @@ private void failPendingMessages(ClientCnx cnx,
PulsarClientException ex) {
ReferenceCountUtil.safeRelease(op.cmd);
op.recycle();
});
-semaphore.release(releaseCount.get());
+
pendingMessages.clear();
pendingCallbacks.clear();
if (isBatchMessagingEnabled()) {
failPendingBatchMessages(ex);
}
+
+semaphore.drainPermits();
+semaphore.release(conf.getMaxPendingMessages());
Review comment:
Do you want to ‘clear’ or ‘reset’ the semaphore here? I think we can't
‘clear’ the semaphore here, because acquire semaphore don't get the producer
instance lock, if others is already acquired, here will ‘release’ it and it
also might lead the dead lock right?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] codelipenghui commented on a change in pull request #5587: [Issue 5585][pulsar-client] Fix producer Semaphore release error
codelipenghui commented on a change in pull request #5587: [Issue
5585][pulsar-client] Fix producer Semaphore release error
URL: https://github.com/apache/pulsar/pull/5587#discussion_r343687907
##
File path:
pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
##
@@ -795,7 +795,7 @@ void ackReceived(ClientCnx cnx, long sequenceId, long
ledgerId, long entryId) {
log.debug("[{}] [{}] Received ack for msg {} ", topic,
producerName, sequenceId);
}
pendingMessages.remove();
-semaphore.release(op.numMessagesInBatch);
+semaphore.release(isBatchMessagingEnabled() ?
op.numMessagesInBatch : 1);
Review comment:
Can you add a method for get the semaphore release count? i see many places
can use it.
it can be used by `op.getSemaphoreReleaseCount()`? or some others
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] codelipenghui edited a comment on issue #5587: [Issue 5585][pulsar-client] Fix producer Semaphore release error
codelipenghui edited a comment on issue #5587: [Issue 5585][pulsar-client] Fix producer Semaphore release error URL: https://github.com/apache/pulsar/pull/5587#issuecomment-551107084 Please try to add some unit tests to ensure this change works well, i think you can try to simulate a producer like replicator do and some replicate messages. so that we can reproduce the problem. And the tests will useful for some future changes which can break again. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] codelipenghui commented on issue #5587: [Issue 5585][pulsar-client] Fix producer Semaphore release error
codelipenghui commented on issue #5587: [Issue 5585][pulsar-client] Fix producer Semaphore release error URL: https://github.com/apache/pulsar/pull/5587#issuecomment-551107084 Please try to add some unit tests to ensure this change works well, i think you can try to simulate a producer like replicator do and some replicate messages. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] cdbartholomew commented on issue #5579: [broker] Every time PersistentMessageExpiryMonitor runs, it deletes a non-expired message when using TTL
cdbartholomew commented on issue #5579: [broker] Every time
PersistentMessageExpiryMonitor runs, it deletes a non-expired message when
using TTL
URL: https://github.com/apache/pulsar/issues/5579#issuecomment-551103559
I believe this also affects the REST expiry call:
`https://pulsar.apache.org/admin/v2/persistent/{tenant}/{namespace}/{topic}/subscription/{subName}/expireMessages/{expireTimeInSeconds}`
Every time I call that API, it removes a message from the topic, even if
`expireTimeInSeconds `is 1.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] KannarFr commented on issue #5519: Create producer issues using function worker as separate node
KannarFr commented on issue #5519: Create producer issues using function worker as separate node URL: https://github.com/apache/pulsar/issues/5519#issuecomment-551084766 I think because the exception occurs instantly. The ZK timeout is confired to 3ms everywhere. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] KannarFr removed a comment on issue #5519: Create producer issues using function worker as separate node
KannarFr removed a comment on issue #5519: Create producer issues using function worker as separate node URL: https://github.com/apache/pulsar/issues/5519#issuecomment-551083392 I think this is related too. ``` 13:41:14.539 [pulsar-external-listener-3-1] ERROR org.apache.pulsar.client.impl.ConsumerImpl - [persistent://clevercloud/accesslogs/clevercloud-adc-n4][clevercloud/functions/haproxyFunction] Message listener error in processing message: 90093:15895:-1 java.lang.RuntimeException: java.lang.InterruptedException at org.apache.pulsar.io.core.PushSource.consume(PushSource.java:70) ~[java-instance.jar:2.4.0] at org.apache.pulsar.functions.source.PulsarSource.received(PulsarSource.java:129) ~[java-instance.jar:2.4.0] at org.apache.pulsar.client.impl.ConsumerImpl.lambda$triggerListener$6(ConsumerImpl.java:864) ~[java-instance.jar:2.4.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_192] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_192] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_192] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_192] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_192] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_192] at org.apache.pulsar.functions.runtime.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [java-instance.jar:2.4.0] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192] Caused by: java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~[?:1.8.0_192] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) ~[?:1.8.0_192] at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350) ~[?:1.8.0_192] at org.apache.pulsar.io.core.PushSource.consume(PushSource.java:68) ~[java-instance.jar:2.4.0] ... 10 more ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] KannarFr removed a comment on issue #5519: Create producer issues using function worker as separate node
KannarFr removed a comment on issue #5519: Create producer issues using function worker as separate node URL: https://github.com/apache/pulsar/issues/5519#issuecomment-549337217 @sijie ^ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] KannarFr edited a comment on issue #5519: Create producer issues using function worker as separate node
KannarFr edited a comment on issue #5519: Create producer issues using function worker as separate node URL: https://github.com/apache/pulsar/issues/5519#issuecomment-551083392 I think this is related too. ``` 13:41:14.539 [pulsar-external-listener-3-1] ERROR org.apache.pulsar.client.impl.ConsumerImpl - [persistent://clevercloud/accesslogs/clevercloud-adc-n4][clevercloud/functions/haproxyFunction] Message listener error in processing message: 90093:15895:-1 java.lang.RuntimeException: java.lang.InterruptedException at org.apache.pulsar.io.core.PushSource.consume(PushSource.java:70) ~[java-instance.jar:2.4.0] at org.apache.pulsar.functions.source.PulsarSource.received(PulsarSource.java:129) ~[java-instance.jar:2.4.0] at org.apache.pulsar.client.impl.ConsumerImpl.lambda$triggerListener$6(ConsumerImpl.java:864) ~[java-instance.jar:2.4.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_192] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_192] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_192] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_192] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_192] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_192] at org.apache.pulsar.functions.runtime.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [java-instance.jar:2.4.0] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192] Caused by: java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~[?:1.8.0_192] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) ~[?:1.8.0_192] at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350) ~[?:1.8.0_192] at org.apache.pulsar.io.core.PushSource.consume(PushSource.java:68) ~[java-instance.jar:2.4.0] ... 10 more ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] KannarFr edited a comment on issue #5519: Create producer issues using function worker as separate node
KannarFr edited a comment on issue #5519: Create producer issues using function worker as separate node URL: https://github.com/apache/pulsar/issues/5519#issuecomment-551083392 ``` 13:41:14.539 [pulsar-external-listener-3-1] ERROR org.apache.pulsar.client.impl.ConsumerImpl - [persistent://clevercloud/accesslogs/clevercloud-adc-n4][clevercloud/functions/haproxyFunction] Message listener error in processing message: 90093:15895:-1 java.lang.RuntimeException: java.lang.InterruptedException at org.apache.pulsar.io.core.PushSource.consume(PushSource.java:70) ~[java-instance.jar:2.4.0] at org.apache.pulsar.functions.source.PulsarSource.received(PulsarSource.java:129) ~[java-instance.jar:2.4.0] at org.apache.pulsar.client.impl.ConsumerImpl.lambda$triggerListener$6(ConsumerImpl.java:864) ~[java-instance.jar:2.4.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_192] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_192] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_192] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_192] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_192] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_192] at org.apache.pulsar.functions.runtime.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [java-instance.jar:2.4.0] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192] Caused by: java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~[?:1.8.0_192] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) ~[?:1.8.0_192] at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350) ~[?:1.8.0_192] at org.apache.pulsar.io.core.PushSource.consume(PushSource.java:68) ~[java-instance.jar:2.4.0] ... 10 more ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] KannarFr removed a comment on issue #5519: Create producer issues using function worker as separate node
KannarFr removed a comment on issue #5519: Create producer issues using function worker as separate node URL: https://github.com/apache/pulsar/issues/5519#issuecomment-549337184 I think the main problem comes from: ``` Nov 01 12:12:57 clevercloud-zookeeper-c2-n3 pulsar-zookeeper[8351]: 12:12:57.482 [ProcessThread(sid:3 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x1054963969b type:delete cxid:0xfdf6 zxid:0x496cb3 txntype:-1 reqpath:n/a Error Path:/ledgers/00/0008 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0008 Nov 01 12:12:57 clevercloud-zookeeper-c2-n3 pulsar-zookeeper[8351]: 12:12:57.503 [ProcessThread(sid:3 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x305496394510002 type:delete cxid:0xf94d zxid:0x496cb5 txntype:-1 reqpath:n/a Error Path:/ledgers/00/0008 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0008 Nov 01 12:12:57 clevercloud-zookeeper-c2-n3 pulsar-zookeeper[8351]: 12:12:57.530 [ProcessThread(sid:3 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x305496394510002 type:delete cxid:0xf950 zxid:0x496cb8 txntype:-1 reqpath:n/a Error Path:/ledgers/00/0008 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0008 Nov 01 12:12:57 clevercloud-zookeeper-c2-n3 pulsar-zookeeper[8351]: 12:12:57.554 [ProcessThread(sid:3 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x305496394510002 type:delete cxid:0xf953 zxid:0x496cbb txntype:-1 reqpath:n/a Error Path:/ledgers/00/0008 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0008 Nov 01 12:12:57 clevercloud-zookeeper-c2-n3 pulsar-zookeeper[8351]: 12:12:57.580 [ProcessThread(sid:3 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x1054963969b type:delete cxid:0xfdfa zxid:0x496cbf txntype:-1 reqpath:n/a Error Path:/ledgers/00/0008 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0008 Nov 01 12:12:57 clevercloud-zookeeper-c2-n3 pulsar-zookeeper[8351]: 12:12:57.600 [ProcessThread(sid:3 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x1054963969b type:delete cxid:0xfdfc zxid:0x496cc1 txntype:-1 reqpath:n/a Error Path:/ledgers/00/0008 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0008 Nov 01 12:12:57 clevercloud-zookeeper-c2-n3 pulsar-zookeeper[8351]: 12:12:57.623 [ProcessThread(sid:3 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x305496394510002 type:delete cxid:0xf956 zxid:0x496cc4 txntype:-1 reqpath:n/a Error Path:/ledgers/00/0008 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0008 Nov 01 12:14:53 clevercloud-zookeeper-c2-n3 pulsar-zookeeper[8351]: 12:14:53.305 [ProcessThread(sid:3 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x1054b6c97d10006 type:create cxid:0x1 zxid:0x496cda txntype:-1 reqpath:n/a Error Path:/pulsar/functions Error:KeeperErrorCode = NodeExists for /pulsar/functions ``` I got it again today, on a ZK node. The broker configured timeout for ZK is 30s, but when I got the exception of sessiontimeout, it's instant, there is no 30s waited to throw the exception. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] KannarFr commented on issue #5519: Create producer issues using function worker as separate node
KannarFr commented on issue #5519: Create producer issues using function worker as separate node URL: https://github.com/apache/pulsar/issues/5519#issuecomment-551083392 ```13:40:44.745 [pulsar-external-listener-3-1] ERROR org.apache.pulsar.client.impl.ConsumerImpl - [persistent://clevercloud/accesslogs/par-adc-n14][clevercloud/functions/haproxyFunction] Message listener error in processing message: 90076:3:-1 java.lang.RuntimeException: java.lang.InterruptedException at org.apache.pulsar.io.core.PushSource.consume(PushSource.java:70) ~[java-instance.jar:2.4.0] at org.apache.pulsar.functions.source.PulsarSource.received(PulsarSource.java:129) ~[java-instance.jar:2.4.0] at org.apache.pulsar.client.impl.ConsumerImpl.lambda$triggerListener$6(ConsumerImpl.java:864) ~[java-instance.jar:2.4.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_192] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_192] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_192] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_192] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_192] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_192] at org.apache.pulsar.functions.runtime.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [java-instance.jar:2.4.0] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192] Caused by: java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~[?:1.8.0_192] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) ~[?:1.8.0_192] at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350) ~[?:1.8.0_192] at org.apache.pulsar.io.core.PushSource.consume(PushSource.java:68) ~[java-instance.jar:2.4.0] ... 10 more ``` I think this is related too. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] sijie opened a new issue #5588: java.lang.IllegalArgumentException: ledgerId doesn't match with acked ledgerId
sijie opened a new issue #5588: java.lang.IllegalArgumentException: ledgerId doesn't match with acked ledgerId URL: https://github.com/apache/pulsar/issues/5588 **Describe the bug** Broker throws IllegalArgumentException on write failures. ``` java.lang.IllegalArgumentException: ledgerId 40 doesn't match with acked ledgerId 39 at com.google.common.base.Preconditions.checkArgument(Preconditions.java:323) ~[com.google.guava-guava-21.0.jar:?] at org.apache.bookkeeper.mledger.impl.OpAddEntry.addComplete(OpAddEntry.java:116) ~[org.apache.pulsar-managed-ledger-original-2.4.1.jar:2.4.1] at org.apache.bookkeeper.client.AsyncCallback$AddCallback.addCompleteWithLatency(AsyncCallback.java:91) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.PendingAddOp.submitCallback(PendingAddOp.java:390) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.LedgerHandle.errorOutPendingAdds(LedgerHandle.java:1772) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.LedgerHandle$5.safeRun(LedgerHandle.java:572) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222] ``` **To Reproduce** Has seen this behavior on several stress tests. **Expected behavior** No exception throw during stress tests. **Additional context** The problem was introduced because we reuse OpAddEntry between two different ledgers. 1) when a ledger fails to write an entry, the ledger is closed. but we didn't clear `pendingAddEntries` in managed ledger. (https://github.com/apache/pulsar/blob/master/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java#L228） 2) when a new ledger is opened, the `pendingAddEntries` will be re-sent by setting to the new ledger handle. (https://github.com/apache/pulsar/blob/master/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L1299) If the callback of an `OpAddEntry` is triggered after the op is already re-sent to the new ledger, it will check if the ledger id in the callback is same as the ledger id in the `OpAddEntry` is same. If the ledger doesn't match, it will throw IllegalArgumentException (https://github.com/apache/pulsar/blob/master/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java#L179) Since we are using recycler for managing OpAddEntry, we should avoid using same ops between different ledger handles. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] weishuisheng opened a new pull request #5587: [Issue 5585][pulsar-client] Fix producer Semaphore release error
weishuisheng opened a new pull request #5587: [Issue 5585][pulsar-client] Fix producer Semaphore release error URL: https://github.com/apache/pulsar/pull/5587 Fixes #5585 fix the Semaphore release in ack This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar-translation] Anonymitaet commented on a change in pull request #7: update schedule
Anonymitaet commented on a change in pull request #7: update schedule URL: https://github.com/apache/pulsar-translation/pull/7#discussion_r343587721 ## File path: schedule.md ## @@ -41,3 +41,25 @@ io-redis | Bu Xing | | In Progress io-solr | Bu Xing | | In Progress io-tcp | Bu Xing | | In Progress security-extending | bilahepan(Tianci Gao) | | In Progress +getting-started-standalone（对应Run Pulsar locally） | | | To Do Review comment: could you please use English so that translators from worldwide can understand This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar-translation] Anonymitaet commented on a change in pull request #7: update schedule
Anonymitaet commented on a change in pull request #7: update schedule URL: https://github.com/apache/pulsar-translation/pull/7#discussion_r343596593 ## File path: schedule.md ## @@ -41,3 +41,25 @@ io-redis | Bu Xing | | In Progress io-solr | Bu Xing | | In Progress io-tcp | Bu Xing | | In Progress security-extending | bilahepan(Tianci Gao) | | In Progress +getting-started-standalone（对应Run Pulsar locally） | | | To Do +getting-started-docker | | | To Do +getting-started-clients | | | To Do +schema-get-started | | | To Do +schema-understand | | | To Do +schema-evolution-compatibility | | | To Do +schema-manage | | | To Do +functions-overview | | | To Do +functions-worker | | | To Do +functions-runtime | | | To Do +functions-develop | | | To Do +functions-debug | | | To Do +functions-deploy | | | To Do +functions-cli | | | To Do +io-overview | | | To Do +io-quickstart（对应 get started） | | | To Do +io-use | | | To Do +io-debug | | | To Do +io-connectors | | | To Do +io-cdc | | | To Do +io-develop | | | To Do +io-cli | | | To Do Review comment: Could you please check the master branch and add the following files (except ones with strikethrough)?  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] sijie commented on issue #5386: Update presto to 0.226
sijie commented on issue #5386: Update presto to 0.226 URL: https://github.com/apache/pulsar/pull/5386#issuecomment-551031933 ``` java.lang.RuntimeException: java.lang.UnsupportedOperationException: com.facebook.presto.spi.block.LongArrayBlock at org.apache.pulsar.sql.presto.PulsarSplitManager.getSplits(PulsarSplitManager.java:135) at org.apache.pulsar.sql.presto.TestPulsarSplitManager.testPublishTimePredicatePushdown(TestPulsarSplitManager.java:206) at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124) at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:54) at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:44) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.UnsupportedOperationException: com.facebook.presto.spi.block.LongArrayBlock at com.facebook.presto.spi.block.Block.getLong(Block.java:72) at org.apache.pulsar.sql.presto.PulsarSplitManager$PredicatePushdownInfo.getPredicatePushdownInfo(PulsarSplitManager.java:356) at org.apache.pulsar.sql.presto.PulsarSplitManager.getSplitsForTopic(PulsarSplitManager.java:257) at org.apache.pulsar.sql.presto.PulsarSplitManager.getSplitsNonPartitionedTopic(PulsarSplitManager.java:228) at org.apache.pulsar.sql.presto.TestPulsarSplitManager$ResultCaptor.answer(TestPulsarSplitManager.java:70) at org.apache.pulsar.sql.presto.PulsarSplitManager.getSplits(PulsarSplitManager.java:127) ``` @aahmed-se you need to fix the code when upgrading to a newer version. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] tuteng edited a comment on issue #5514: use pulsar get error in mac os
tuteng edited a comment on issue #5514: use pulsar get error in mac os URL: https://github.com/apache/pulsar/issues/5514#issuecomment-551026974 There seems to be a lack of package boost-python3, if you try to install with source code, don't install with pip anymore. Python3 is no problem, you don't need to reinstall it. The following command is installed based on source code: ``` git clone https://github.com/apache/pulsar cd pulsar git checkout branch-2.4 brew install boost-python3 cd pulsar-client-cpp cmake . make -j4 && make install cd python python setup.py install # python Python 3.7.3 (default, Mar 27 2019, 09:23:32) [Clang 9.0.0 (clang-900.0.39.2)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import pulsar >>> dir(pulsar) ['Authentication', 'AuthenticationAthenz', 'AuthenticationTLS', 'AuthenticationToken', 'Client', 'CompressionType', 'Consumer', 'ConsumerType', 'Context', 'Function', 'IdentitySerDe', 'InitialPosition', 'Message', 'MessageBatch', 'MessageId', 'PartitionsRoutingMode', 'PickleSerDe', 'Producer', 'Reader', 'Result', 'SerDe', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '_check_type', '_check_type_or_none', '_listener_wrapper', '_pulsar', '_retype', '_schema', 'certifi', 'functions', 're', 'schema'] >>> ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] tuteng commented on issue #5514: use pulsar get error in mac os
tuteng commented on issue #5514: use pulsar get error in mac os URL: https://github.com/apache/pulsar/issues/5514#issuecomment-551026974 There seems to be a lack of package boost-python3, if you try to install with source code, don't install with pip anymore. The following command is installed based on source code: ``` git clone https://github.com/apache/pulsar cd pulsar git checkout branch-2.4 brew install boost-python3 cd pulsar-client-cpp cmake . make -j4 && make install cd python python setup.py install # python Python 3.7.3 (default, Mar 27 2019, 09:23:32) [Clang 9.0.0 (clang-900.0.39.2)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import pulsar >>> dir(pulsar) ['Authentication', 'AuthenticationAthenz', 'AuthenticationTLS', 'AuthenticationToken', 'Client', 'CompressionType', 'Consumer', 'ConsumerType', 'Context', 'Function', 'IdentitySerDe', 'InitialPosition', 'Message', 'MessageBatch', 'MessageId', 'PartitionsRoutingMode', 'PickleSerDe', 'Producer', 'Reader', 'Result', 'SerDe', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '_check_type', '_check_type_or_none', '_listener_wrapper', '_pulsar', '_retype', '_schema', 'certifi', 'functions', 're', 'schema'] >>> ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[pulsar] branch master updated (cd259a4 -> 45bb182)
This is an automated email from the ASF dual-hosted git repository. aahmed pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from cd259a4 Fix schema static initialization sequence (#5577) add 45bb182 Fix license file in the distribution package (#5578) No new revisions were added by this update. Summary of changes: distribution/server/src/assemble/LICENSE.bin.txt | 7 ++--- pulsar-sql/presto-distribution/LICENSE | 33 2 files changed, 20 insertions(+), 20 deletions(-)
[GitHub] [pulsar] aahmed-se commented on issue #5576: Remove old Pulsar website
aahmed-se commented on issue #5576: Remove old Pulsar website URL: https://github.com/apache/pulsar/pull/5576#issuecomment-551026499 run java8 tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] aahmed-se merged pull request #5578: Fix license file in the distribution package
aahmed-se merged pull request #5578: Fix license file in the distribution package URL: https://github.com/apache/pulsar/pull/5578 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] aahmed-se merged pull request #5577: Fix schema static initialization sequence
aahmed-se merged pull request #5577: Fix schema static initialization sequence URL: https://github.com/apache/pulsar/pull/5577 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[pulsar] branch master updated (4d0983a -> cd259a4)
This is an automated email from the ASF dual-hosted git repository. aahmed pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 4d0983a [Issue 5541] cpp and python API: consumer and reader seek (#5542) add cd259a4 Fix schema static initialization sequence (#5577) No new revisions were added by this update. Summary of changes: .../pulsar/client/impl/schema/BooleanSchema.java | 8 +++ .../pulsar/client/impl/schema/ByteBufSchema.java | 8 +++ .../client/impl/schema/ByteBufferSchema.java | 8 +++ .../pulsar/client/impl/schema/ByteSchema.java | 8 +++ .../pulsar/client/impl/schema/BytesSchema.java | 7 +++--- .../pulsar/client/impl/schema/DateSchema.java | 7 +++--- .../pulsar/client/impl/schema/DoubleSchema.java| 7 +++--- .../pulsar/client/impl/schema/FloatSchema.java | 8 +++ .../pulsar/client/impl/schema/IntSchema.java | 8 +++ .../pulsar/client/impl/schema/LongSchema.java | 8 +++ .../pulsar/client/impl/schema/ShortSchema.java | 8 +++ .../pulsar/client/impl/schema/TimeSchema.java | 7 +++--- .../pulsar/client/impl/schema/TimestampSchema.java | 7 +++--- .../client/impl/schema/PrimitiveSchemaTest.java| 27 ++ 14 files changed, 68 insertions(+), 58 deletions(-)
[GitHub] [pulsar-translation] Jennifer88huang edited a comment on issue #7: update schedule
Jennifer88huang edited a comment on issue #7: update schedule URL: https://github.com/apache/pulsar-translation/pull/7#issuecomment-551018195 @SylviaBABY You can learn the description content from https://github.com/apache/pulsar/pulls The following is an example: https://github.com/apache/pulsar/pull/5512 Please keep only necessary info in the description. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] congbobo184 removed a comment on issue #5570: Transaction log implemention
congbobo184 removed a comment on issue #5570: Transaction log implemention URL: https://github.com/apache/pulsar/pull/5570#issuecomment-551018258 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] congbobo184 commented on issue #5570: Transaction log implemention
congbobo184 commented on issue #5570: Transaction log implemention URL: https://github.com/apache/pulsar/pull/5570#issuecomment-551018258 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar-translation] Jennifer88huang commented on issue #7: update schedule
Jennifer88huang commented on issue #7: update schedule URL: https://github.com/apache/pulsar-translation/pull/7#issuecomment-551018195 @SylviaBABY You can learn the description content from https://github.com/apache/pulsar/pull/5512 Please keep only necessary info in the description. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar-translation] Jennifer88huang commented on issue #7: update schedule
Jennifer88huang commented on issue #7: update schedule URL: https://github.com/apache/pulsar-translation/pull/7#issuecomment-551017218 @Anonymitaet could you please help review this PR? Thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] codelipenghui edited a comment on issue #5388: [PIP-51] Introduce sticky consumer
codelipenghui edited a comment on issue #5388: [PIP-51] Introduce sticky consumer URL: https://github.com/apache/pulsar/pull/5388#issuecomment-551016454 @sijie I have added a task tracker #5586 to track the reset cursor approach for sticky consumer. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] codelipenghui commented on issue #5388: [PIP-51] Introduce sticky consumer
codelipenghui commented on issue #5388: [PIP-51] Introduce sticky consumer URL: https://github.com/apache/pulsar/pull/5388#issuecomment-551016454 @sijie I have added a task tracker to track the reset cursor approach for sticky consumer. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] codelipenghui opened a new issue #5586: [DISCUSS] Sticky consumer reset cursor
codelipenghui opened a new issue #5586: [DISCUSS] Sticky consumer reset cursor URL: https://github.com/apache/pulsar/issues/5586 **Is your feature request related to a problem? Please describe.** Sticky consumer consume messages with distributed to a fixed hash key range, also like a logical partition or a virtual partition. PR https://github.com/apache/pulsar/pull/5388 is added support for sticky consumer. However, we need to rethink the reset cursor of sticky consumer. Currently, if one sticky consumer reset cursor to a historical point, other sticky consumers cursor also be set. So it's better to explore a new approach for sticky consumer reset cursor which can avoid effect between sticky consumers. Added issue to track the approach of sticky consumer reset cursor, if anyone has ideas please add comments here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] codelipenghui commented on issue #5585: Non-persistent topic's replication has a deadlock
codelipenghui commented on issue #5585: Non-persistent topic's replication has a deadlock URL: https://github.com/apache/pulsar/issues/5585#issuecomment-551011578 welcome @weishuisheng , look forward to your fix. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] weishuisheng opened a new issue #5585: Non-persistent topic's replication has a deadlock
weishuisheng opened a new issue #5585: Non-persistent topic's replication has a deadlock URL: https://github.com/apache/pulsar/issues/5585 NonPersistentReplicator disable batching. If there is batch message,producer will acquire one,but release num in message's meta when process ack. ``` // When publishing during replication, we need to set the correct number of message in batch // This is only used in tracking the publish rate stats int numMessages = msg.getMessageBuilder().hasNumMessagesInBatch() ? msg.getMessageBuilder().getNumMessagesInBatch() : 1; ByteBufPair cmd = sendMessage(producerId, sequenceId, numMessages, msgMetadata, encryptedPayload); msgMetadataBuilder.recycle(); msgMetadata.recycle(); final OpSendMsg op = OpSendMsg.create(msg, cmd, sequenceId, callback); op.setNumMessagesInBatch(numMessages); op.setBatchSizeByte(encryptedPayload.readableBytes()); pendingMessages.put(op); lastSendFuture = callback.getFuture(); ``` So the ProducerImpl's semaphore no longer has any effect, the ProducerImpl's sendAsync method maybe blocked in pendingMessages's put method . There may be a deadlock between ackReceived and sendAsync . The deadlock is between pulsar-io-22-8 and pulsar-io-22-12 in jstack log. [broker jstack log](https://github.com/apache/pulsar/files/3818922/broker.txt) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] Jennifer88huang commented on issue #5520: Missing documentation for window functions
Jennifer88huang commented on issue #5520: Missing documentation for window functions URL: https://github.com/apache/pulsar/issues/5520#issuecomment-551007815 @srkukarni could you help add the related documentation? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar-translation] SylviaBABY opened a new pull request #7: update schedule
SylviaBABY opened a new pull request #7: update schedule URL: https://github.com/apache/pulsar-translation/pull/7 <-- ### Contribution Checklist - Name the pull request in the form "[Issue XYZ][component] Title of the pull request", where *XYZ* should be replaced by the actual issue number. Skip *Issue XYZ* if there is no associated github issue for this pull request. Skip *component* if you are unsure about which is the best component. E.g. `[docs] Fix typo in produce method`. - Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review. - Each pull request should address only one issue, not mix up code from multiple issues. - Each commit in the pull request has a meaningful commit message - Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below. **(The sections below can be removed for hotfixes of typos)** --> *(If this PR fixes a github issue, please add `Fixes #`.)* Fixes # *(or if this PR is one task of a github issue, please add `Master Issue: #` to link to the master issue.)* Master Issue: # ### Motivation *Explain here the context, and why you're making that change. What is the problem you're trying to solve.* ### Modifications *Describe the modifications you've done.* ### Verify this change - [ ] Make sure that the change is correct. - For how to check and verify, refer to [Translation and localization](https://github.com/apache/pulsar/tree/master/site2#translation-and-localization). ### Documentation - Is this pull request related to crowdin usage? (yes / no) - Is this pull request related to translation quality? (yes / no) - If yes, how to improve? - Is this pull request related to translation guidelines?(yes / no) - Is this pull request related to translation workflow?(yes / no) - If a sth is not documented yet in this PR, please create a followup issue for adding the documentation. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] Anonymitaet commented on issue #5584: [Doc] Improve Pulsar Functions Admin API guide
Anonymitaet commented on issue #5584: [Doc] Improve Pulsar Functions Admin API guide URL: https://github.com/apache/pulsar/issues/5584#issuecomment-550987073 @wolfstudy could you please provide technical inputs? Thank you This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] Anonymitaet opened a new issue #5584: [Doc] Improve Pulsar Functions Admin API guide
Anonymitaet opened a new issue #5584: [Doc] Improve Pulsar Functions Admin API guide URL: https://github.com/apache/pulsar/issues/5584 Currently, Pulsar Functions Admin API is [here](https://github.com/apache/pulsar/blob/master/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/Functions.java) but not well documented. Besides, @sijie suggests adding examples. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
