date:20201113

[GitHub] [pulsar] eolivelli commented on a change in pull request #8564: [pulsar-broker] Fix topic policy update

2020-11-13 Thread GitBox



eolivelli commented on a change in pull request #8564:
URL: https://github.com/apache/pulsar/pull/8564#discussion_r523390675



##
File path: 
pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiTest2.java
##
@@ -102,6 +102,8 @@
 @Override
 public void setup() throws Exception {
 conf.setLoadBalancerEnabled(true);
+conf.setTopicLevelPoliciesEnabled(true);
+conf.setSystemTopicEnabled(true);

Review comment:
   This change is affecting all of the other tests in this file.
   Do we have some side effect?
   Are we going to use more and more resources for al the tests in this file?
   My question is more general, sometimes we enable feature in common setup 
utilities, with the possibility to alter the semantics of the other tests.
   
   Probably in this case there is no issue. Please clarify, I am still new to 
this suite of tests





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] gaoran10 commented on pull request #8563: [Transaction] Guarantee transaction metadata handlers connect

2020-11-13 Thread GitBox



gaoran10 commented on pull request #8563:
URL: https://github.com/apache/pulsar/pull/8563#issuecomment-727149745


   /pulsarbot run-failure-checks



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] rdhabalia commented on pull request #8565: [pulsar-broker] Fix: Topic dedup policy configuration doesn't support V1 topic-name

2020-11-13 Thread GitBox



rdhabalia commented on pull request #8565:
URL: https://github.com/apache/pulsar/pull/8565#issuecomment-727143221


   /pulsarbot run-failure-checks



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] rdhabalia commented on pull request #8564: [pulsar-broker] Fix topic policy update

2020-11-13 Thread GitBox



rdhabalia commented on pull request #8564:
URL: https://github.com/apache/pulsar/pull/8564#issuecomment-727139620


   /pulsarbot run-failure-checks



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] codelipenghui commented on pull request #8549: Issue 6478: Fix the misleading setting in presto configuration

2020-11-13 Thread GitBox



codelipenghui commented on pull request #8549:
URL: https://github.com/apache/pulsar/pull/8549#issuecomment-727123412


   /pulsarbot run-failure-checks



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] gaoran10 commented on pull request #8563: [Transaction] Guarantee transaction metadata handlers connect

2020-11-13 Thread GitBox



gaoran10 commented on pull request #8563:
URL: https://github.com/apache/pulsar/pull/8563#issuecomment-727123031


   /pulsarbot run-failure-checks



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] codelipenghui commented on pull request #8544: Consumer filtering

2020-11-13 Thread GitBox



codelipenghui commented on pull request #8544:
URL: https://github.com/apache/pulsar/pull/8544#issuecomment-727122863


   @andrekramer1 We should avoid repacking the batch at the broker side. This 
will introduce more GC workload and need more CPU cycles to repack the message. 
You can see 
https://github.com/apache/pulsar/wiki/PIP-70%3A-Introduce-lightweight-raw-Message-metadata,
 try to avoid serialize and deserialize the message again on the server-side.
   
   I want to describe my rough idea, If it is a batch message, we have message 
metadata for the batch and single message metadata for each message. So I think 
we can add all tags into the message metadata of the batch. For example, 2 
messages in a batch and property1 for message0 is 1, property1 for message1 is 
2. So that we can add property1 = [1,2] in the batch message metadata. So, the 
broker can filter out the batches. 
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] rdhabalia commented on pull request #8565: [pulsar-broker] Fix: Topic dedup policy configuration doesn't support V1 topic-name

2020-11-13 Thread GitBox



rdhabalia commented on pull request #8565:
URL: https://github.com/apache/pulsar/pull/8565#issuecomment-727120463


   /pulsarbot run-failure-checks



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] hangc0276 commented on issue #8502: Upgrade Debezium to a newer version

2020-11-13 Thread GitBox



hangc0276 commented on issue #8502:
URL: https://github.com/apache/pulsar/issues/8502#issuecomment-727100981


   @arumugamtp  Are you trying to fix it?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] rdhabalia opened a new pull request #8565: [pulsar-broker] Fix: Topic dedup policy configuration doesn't support V1 topic-name

2020-11-13 Thread GitBox



rdhabalia opened a new pull request #8565:
URL: https://github.com/apache/pulsar/pull/8565


   ### Motivation
   Right now, Topic level dedup policy configuration doesn't support V1 topic 
with cluster name in it.
   
   **NOTE:** This is PR is on top of: #8555



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] rdhabalia opened a new pull request #8564: [pulsar-broker] Fix topic policy update

2020-11-13 Thread GitBox



rdhabalia opened a new pull request #8564:
URL: https://github.com/apache/pulsar/pull/8564


   ### Motivation
   Right now, TopicPolicy doesn't have auto refresh mechanism and trigger of 
refresh is admin api change. Now, enabling dedup admin api call can go to any 
broker and owner broker doesn't refresh and apply the change. Also, listening 
to all topics for every broker might not be scalable so, let redirect to broker 
which owns the topic and update the topic policy cache.
   
   ### Modification
   redirect and let topic owner broker to enable dedup and refresh the topic 
policy cache.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[pulsar] branch master updated (99e9543 -> 9aaa1cc)

2020-11-13 Thread sijie

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git.


from 99e9543  [Tests] Avoid use Thread.sleep() in TopicPoliciesTest (#8557)
 add 9aaa1cc  [Functions] Refactor Context and State API to allow plugging 
different state store implementations (#8537)

No new revisions were added by this update.

Summary of changes:
 .../org/apache/pulsar/client/api/Consumer.java |   3 +-
 .../org/apache/pulsar/client/api/Producer.java |   3 +-
 .../org/apache/pulsar/functions/api/Context.java   |  37 -
 .../api/{Function.java => StateStore.java} |  59 +--
 .../api/{SerDe.java => StateStoreContext.java} |  16 +-
 .../functions/api/state/ByteBufferStateStore.java  |  76 +
 .../functions/api/state/CounterStateStore.java |  63 +++
 .../pulsar/functions/api/state}/package-info.java  |   4 +-
 .../pulsar/functions/instance/ContextImpl.java | 112 +++--
 .../functions/instance/JavaInstanceRunnable.java   | 179 +---
 ...StateContextImpl.java => BKStateStoreImpl.java} | 108 ++--
 .../instance/state/BKStateStoreProviderImpl.java   | 182 +
 .../instance/state/DefaultStateStore.java  |  17 +-
 .../instance/state/InstanceStateManager.java   |  81 +
 .../functions/instance/state/StateContext.java |  78 -
 .../functions/instance/state/StateManager.java |  30 ++--
 .../instance/state/StateStoreContextImpl.java  |  10 +-
 .../instance/state/StateStoreProvider.java |  70 
 .../pulsar/functions/instance/ContextImplTest.java |  25 +--
 ...textImplTest.java => BKStateStoreImplTest.java} |  33 +++-
 .../instance/state/InstanceStateManagerTest.java   | 122 ++
 .../apache/pulsar/io/core/ConnectorContext.java|  15 ++
 22 files changed, 955 insertions(+), 368 deletions(-)
 copy 
pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/{Function.java
 => StateStore.java} (58%)
 copy 
pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/{SerDe.java
 => StateStoreContext.java} (83%)
 create mode 100644 
pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/state/ByteBufferStateStore.java
 create mode 100644 
pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/state/CounterStateStore.java
 copy {pulsar-common/src/main/java/org/apache/pulsar/common/policies => 
pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/state}/package-info.java
 (92%)
 rename 
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/{StateContextImpl.java
 => BKStateStoreImpl.java} (56%)
 create mode 100644 
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/BKStateStoreProviderImpl.java
 copy 
pulsar-broker-common/src/main/java/org/apache/pulsar/common/configuration/Category.java
 => 
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/DefaultStateStore.java
 (72%)
 create mode 100644 
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/InstanceStateManager.java
 delete mode 100644 
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/StateContext.java
 copy 
pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/ProxyStats.java
 => 
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/StateManager.java
 (56%)
 copy 
pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/Sink.java => 
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/StateStoreContextImpl.java
 (77%)
 create mode 100644 
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/StateStoreProvider.java
 rename 
pulsar-functions/instance/src/test/java/org/apache/pulsar/functions/instance/state/{StateContextImplTest.java
 => BKStateStoreImplTest.java} (80%)
 create mode 100644 
pulsar-functions/instance/src/test/java/org/apache/pulsar/functions/instance/state/InstanceStateManagerTest.java

[pulsar] branch master updated (99e9543 -> 9aaa1cc)

2020-11-13 Thread sijie

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git.


from 99e9543  [Tests] Avoid use Thread.sleep() in TopicPoliciesTest (#8557)
 add 9aaa1cc  [Functions] Refactor Context and State API to allow plugging 
different state store implementations (#8537)

No new revisions were added by this update.

Summary of changes:
 .../org/apache/pulsar/client/api/Consumer.java |   3 +-
 .../org/apache/pulsar/client/api/Producer.java |   3 +-
 .../org/apache/pulsar/functions/api/Context.java   |  37 -
 .../api/{Function.java => StateStore.java} |  59 +--
 .../api/{SerDe.java => StateStoreContext.java} |  16 +-
 .../functions/api/state/ByteBufferStateStore.java  |  76 +
 .../functions/api/state/CounterStateStore.java |  63 +++
 .../pulsar/functions/api/state}/package-info.java  |   4 +-
 .../pulsar/functions/instance/ContextImpl.java | 112 +++--
 .../functions/instance/JavaInstanceRunnable.java   | 179 +---
 ...StateContextImpl.java => BKStateStoreImpl.java} | 108 ++--
 .../instance/state/BKStateStoreProviderImpl.java   | 182 +
 .../instance/state/DefaultStateStore.java  |  17 +-
 .../instance/state/InstanceStateManager.java   |  81 +
 .../functions/instance/state/StateContext.java |  78 -
 .../functions/instance/state/StateManager.java |  30 ++--
 .../instance/state/StateStoreContextImpl.java  |  10 +-
 .../instance/state/StateStoreProvider.java |  70 
 .../pulsar/functions/instance/ContextImplTest.java |  25 +--
 ...textImplTest.java => BKStateStoreImplTest.java} |  33 +++-
 .../instance/state/InstanceStateManagerTest.java   | 122 ++
 .../apache/pulsar/io/core/ConnectorContext.java|  15 ++
 22 files changed, 955 insertions(+), 368 deletions(-)
 copy 
pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/{Function.java
 => StateStore.java} (58%)
 copy 
pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/{SerDe.java
 => StateStoreContext.java} (83%)
 create mode 100644 
pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/state/ByteBufferStateStore.java
 create mode 100644 
pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/state/CounterStateStore.java
 copy {pulsar-common/src/main/java/org/apache/pulsar/common/policies => 
pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/state}/package-info.java
 (92%)
 rename 
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/{StateContextImpl.java
 => BKStateStoreImpl.java} (56%)
 create mode 100644 
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/BKStateStoreProviderImpl.java
 copy 
pulsar-broker-common/src/main/java/org/apache/pulsar/common/configuration/Category.java
 => 
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/DefaultStateStore.java
 (72%)
 create mode 100644 
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/InstanceStateManager.java
 delete mode 100644 
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/StateContext.java
 copy 
pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/ProxyStats.java
 => 
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/StateManager.java
 (56%)
 copy 
pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/Sink.java => 
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/StateStoreContextImpl.java
 (77%)
 create mode 100644 
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/StateStoreProvider.java
 rename 
pulsar-functions/instance/src/test/java/org/apache/pulsar/functions/instance/state/{StateContextImplTest.java
 => BKStateStoreImplTest.java} (80%)
 create mode 100644 
pulsar-functions/instance/src/test/java/org/apache/pulsar/functions/instance/state/InstanceStateManagerTest.java

[GitHub] [pulsar] sijie commented on pull request #8549: Issue 6478: Fix the misleading setting in presto configuration

2020-11-13 Thread GitBox



sijie commented on pull request #8549:
URL: https://github.com/apache/pulsar/pull/8549#issuecomment-727045352


   @Jennifer88huang No. It should impact the releases onwards. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[pulsar] branch master updated (99e9543 -> 9aaa1cc)

2020-11-13 Thread sijie

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git.


from 99e9543  [Tests] Avoid use Thread.sleep() in TopicPoliciesTest (#8557)
 add 9aaa1cc  [Functions] Refactor Context and State API to allow plugging 
different state store implementations (#8537)

No new revisions were added by this update.

Summary of changes:
 .../org/apache/pulsar/client/api/Consumer.java |   3 +-
 .../org/apache/pulsar/client/api/Producer.java |   3 +-
 .../org/apache/pulsar/functions/api/Context.java   |  37 -
 .../api/{Function.java => StateStore.java} |  59 +--
 .../api/{SerDe.java => StateStoreContext.java} |  16 +-
 .../functions/api/state/ByteBufferStateStore.java  |  76 +
 .../functions/api/state/CounterStateStore.java |  63 +++
 .../pulsar/functions/api/state}/package-info.java  |   4 +-
 .../pulsar/functions/instance/ContextImpl.java | 112 +++--
 .../functions/instance/JavaInstanceRunnable.java   | 179 +---
 ...StateContextImpl.java => BKStateStoreImpl.java} | 108 ++--
 .../instance/state/BKStateStoreProviderImpl.java   | 182 +
 .../instance/state/DefaultStateStore.java  |  17 +-
 .../instance/state/InstanceStateManager.java   |  81 +
 .../functions/instance/state/StateContext.java |  78 -
 .../functions/instance/state/StateManager.java |  30 ++--
 .../instance/state/StateStoreContextImpl.java  |  10 +-
 .../instance/state/StateStoreProvider.java |  70 
 .../pulsar/functions/instance/ContextImplTest.java |  25 +--
 ...textImplTest.java => BKStateStoreImplTest.java} |  33 +++-
 .../instance/state/InstanceStateManagerTest.java   | 122 ++
 .../apache/pulsar/io/core/ConnectorContext.java|  15 ++
 22 files changed, 955 insertions(+), 368 deletions(-)
 copy 
pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/{Function.java
 => StateStore.java} (58%)
 copy 
pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/{SerDe.java
 => StateStoreContext.java} (83%)
 create mode 100644 
pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/state/ByteBufferStateStore.java
 create mode 100644 
pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/state/CounterStateStore.java
 copy {pulsar-common/src/main/java/org/apache/pulsar/common/policies => 
pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/state}/package-info.java
 (92%)
 rename 
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/{StateContextImpl.java
 => BKStateStoreImpl.java} (56%)
 create mode 100644 
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/BKStateStoreProviderImpl.java
 copy 
pulsar-broker-common/src/main/java/org/apache/pulsar/common/configuration/Category.java
 => 
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/DefaultStateStore.java
 (72%)
 create mode 100644 
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/InstanceStateManager.java
 delete mode 100644 
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/StateContext.java
 copy 
pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/ProxyStats.java
 => 
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/StateManager.java
 (56%)
 copy 
pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/Sink.java => 
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/StateStoreContextImpl.java
 (77%)
 create mode 100644 
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/StateStoreProvider.java
 rename 
pulsar-functions/instance/src/test/java/org/apache/pulsar/functions/instance/state/{StateContextImplTest.java
 => BKStateStoreImplTest.java} (80%)
 create mode 100644 
pulsar-functions/instance/src/test/java/org/apache/pulsar/functions/instance/state/InstanceStateManagerTest.java

[pulsar] branch master updated: [Functions] Refactor Context and State API to allow plugging different state store implementations (#8537)

2020-11-13 Thread sijie

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 9aaa1cc  [Functions] Refactor Context and State API to allow plugging 
different state store implementations (#8537)
9aaa1cc is described below

commit 9aaa1ccefb3ee1c543e2f0daeb23094cf8773b6a
Author: Sijie Guo 
AuthorDate: Fri Nov 13 14:24:18 2020 -0700

[Functions] Refactor Context and State API to allow plugging different 
state store implementations (#8537)

*Motivation*

Currently, the state API is tied to bookkeeper table implementation. For 
users who already run a database, it
might be useful to allow users to configure to use a different state store 
backend.

This change refactors the context and state API to allow plugging different 
state store implementations
---
 .../org/apache/pulsar/client/api/Consumer.java |   3 +-
 .../org/apache/pulsar/client/api/Producer.java |   3 +-
 .../org/apache/pulsar/functions/api/Context.java   |  37 -
 .../apache/pulsar/functions/api/StateStore.java|  66 
 .../pulsar/functions/api/StateStoreContext.java|  30 
 .../functions/api/state/ByteBufferStateStore.java  |  76 +
 .../functions/api/state/CounterStateStore.java |  63 +++
 .../pulsar/functions/api/state/package-info.java   |  22 +++
 .../pulsar/functions/instance/ContextImpl.java | 112 +++--
 .../functions/instance/JavaInstanceRunnable.java   | 179 +---
 ...StateContextImpl.java => BKStateStoreImpl.java} | 108 ++--
 .../instance/state/BKStateStoreProviderImpl.java   | 182 +
 .../instance/state/DefaultStateStore.java  |  28 
 .../instance/state/InstanceStateManager.java   |  81 +
 .../functions/instance/state/StateContext.java |  78 -
 .../functions/instance/state/StateManager.java |  47 ++
 .../instance/state/StateStoreContextImpl.java  |  27 +++
 .../instance/state/StateStoreProvider.java |  70 
 .../pulsar/functions/instance/ContextImplTest.java |  25 +--
 ...textImplTest.java => BKStateStoreImplTest.java} |  33 +++-
 .../instance/state/InstanceStateManagerTest.java   | 122 ++
 .../apache/pulsar/io/core/ConnectorContext.java|  15 ++
 22 files changed, 1097 insertions(+), 310 deletions(-)

diff --git 
a/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Consumer.java 
b/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Consumer.java
index f1c4f0e..e21f653 100644
--- a/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Consumer.java
+++ b/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Consumer.java
@@ -22,11 +22,10 @@ import java.io.Closeable;
 import java.util.List;
 import java.util.concurrent.CompletableFuture;
 import java.util.concurrent.TimeUnit;
+import org.apache.pulsar.client.api.transaction.Transaction;
 import org.apache.pulsar.common.classification.InterfaceAudience;
 import org.apache.pulsar.common.classification.InterfaceStability;
 
-import org.apache.pulsar.client.api.transaction.Transaction;
-
 /**
  * An interface that abstracts behavior of Pulsar's consumer.
  *
diff --git 
a/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Producer.java 
b/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Producer.java
index 883cf2a..b94a6fe 100644
--- a/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Producer.java
+++ b/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Producer.java
@@ -20,11 +20,10 @@ package org.apache.pulsar.client.api;
 
 import java.io.Closeable;
 import java.util.concurrent.CompletableFuture;
+import org.apache.pulsar.client.api.transaction.Transaction;
 import org.apache.pulsar.common.classification.InterfaceAudience;
 import org.apache.pulsar.common.classification.InterfaceStability;
 
-import org.apache.pulsar.client.api.transaction.Transaction;
-
 /**
  * Producer is used to publish messages on a topic.
  *
diff --git 
a/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Context.java
 
b/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Context.java
index 6fd4dd7..db986a1 100644
--- 
a/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Context.java
+++ 
b/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Context.java
@@ -19,7 +19,10 @@
 package org.apache.pulsar.functions.api;
 
 import java.nio.ByteBuffer;
-
+import java.util.Collection;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
 import org.apache.pulsar.client.api.ConsumerBuilder;
 import org.apache.pulsar.client.api.PulsarClientException;
 import org.apache.pulsar.client.api.Schema;
@@ -28,11 +31,6 @@ import 
org.apache.pulsar.common.cla

[pulsar] branch master updated: [Functions] Refactor Context and State API to allow plugging different state store implementations (#8537)

2020-11-13 Thread sijie

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 9aaa1cc  [Functions] Refactor Context and State API to allow plugging 
different state store implementations (#8537)
9aaa1cc is described below

commit 9aaa1ccefb3ee1c543e2f0daeb23094cf8773b6a
Author: Sijie Guo 
AuthorDate: Fri Nov 13 14:24:18 2020 -0700

[Functions] Refactor Context and State API to allow plugging different 
state store implementations (#8537)

*Motivation*

Currently, the state API is tied to bookkeeper table implementation. For 
users who already run a database, it
might be useful to allow users to configure to use a different state store 
backend.

This change refactors the context and state API to allow plugging different 
state store implementations
---
 .../org/apache/pulsar/client/api/Consumer.java |   3 +-
 .../org/apache/pulsar/client/api/Producer.java |   3 +-
 .../org/apache/pulsar/functions/api/Context.java   |  37 -
 .../apache/pulsar/functions/api/StateStore.java|  66 
 .../pulsar/functions/api/StateStoreContext.java|  30 
 .../functions/api/state/ByteBufferStateStore.java  |  76 +
 .../functions/api/state/CounterStateStore.java |  63 +++
 .../pulsar/functions/api/state/package-info.java   |  22 +++
 .../pulsar/functions/instance/ContextImpl.java | 112 +++--
 .../functions/instance/JavaInstanceRunnable.java   | 179 +---
 ...StateContextImpl.java => BKStateStoreImpl.java} | 108 ++--
 .../instance/state/BKStateStoreProviderImpl.java   | 182 +
 .../instance/state/DefaultStateStore.java  |  28 
 .../instance/state/InstanceStateManager.java   |  81 +
 .../functions/instance/state/StateContext.java |  78 -
 .../functions/instance/state/StateManager.java |  47 ++
 .../instance/state/StateStoreContextImpl.java  |  27 +++
 .../instance/state/StateStoreProvider.java |  70 
 .../pulsar/functions/instance/ContextImplTest.java |  25 +--
 ...textImplTest.java => BKStateStoreImplTest.java} |  33 +++-
 .../instance/state/InstanceStateManagerTest.java   | 122 ++
 .../apache/pulsar/io/core/ConnectorContext.java|  15 ++
 22 files changed, 1097 insertions(+), 310 deletions(-)

diff --git 
a/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Consumer.java 
b/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Consumer.java
index f1c4f0e..e21f653 100644
--- a/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Consumer.java
+++ b/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Consumer.java
@@ -22,11 +22,10 @@ import java.io.Closeable;
 import java.util.List;
 import java.util.concurrent.CompletableFuture;
 import java.util.concurrent.TimeUnit;
+import org.apache.pulsar.client.api.transaction.Transaction;
 import org.apache.pulsar.common.classification.InterfaceAudience;
 import org.apache.pulsar.common.classification.InterfaceStability;
 
-import org.apache.pulsar.client.api.transaction.Transaction;
-
 /**
  * An interface that abstracts behavior of Pulsar's consumer.
  *
diff --git 
a/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Producer.java 
b/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Producer.java
index 883cf2a..b94a6fe 100644
--- a/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Producer.java
+++ b/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Producer.java
@@ -20,11 +20,10 @@ package org.apache.pulsar.client.api;
 
 import java.io.Closeable;
 import java.util.concurrent.CompletableFuture;
+import org.apache.pulsar.client.api.transaction.Transaction;
 import org.apache.pulsar.common.classification.InterfaceAudience;
 import org.apache.pulsar.common.classification.InterfaceStability;
 
-import org.apache.pulsar.client.api.transaction.Transaction;
-
 /**
  * Producer is used to publish messages on a topic.
  *
diff --git 
a/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Context.java
 
b/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Context.java
index 6fd4dd7..db986a1 100644
--- 
a/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Context.java
+++ 
b/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Context.java
@@ -19,7 +19,10 @@
 package org.apache.pulsar.functions.api;
 
 import java.nio.ByteBuffer;
-
+import java.util.Collection;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
 import org.apache.pulsar.client.api.ConsumerBuilder;
 import org.apache.pulsar.client.api.PulsarClientException;
 import org.apache.pulsar.client.api.Schema;
@@ -28,11 +31,6 @@ import 
org.apache.pulsar.common.cla

[GitHub] [pulsar] sijie merged pull request #8537: [Functions] Refactor Context and State API to allow plugging different state store implementations

2020-11-13 Thread GitBox



sijie merged pull request #8537:
URL: https://github.com/apache/pulsar/pull/8537


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] rdhabalia commented on pull request #8544: Consumer filtering

2020-11-13 Thread GitBox



rdhabalia commented on pull request #8544:
URL: https://github.com/apache/pulsar/pull/8544#issuecomment-726994781


   @andrekramer1  can you test and share broker CPU performance graph with this 
change. may be 5K topics with each topic having 2-3 subscriptions and each 
consumer is having 3-5 filters. and it would be great if you can also collect 
other metrics such as : gc, e2e latency for other topics which don't have these 
filters (to validate noisy neighbor scenario). 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] rdhabalia commented on pull request #8555: [pulsar-broker] support v1 topic dedup admin api

2020-11-13 Thread GitBox



rdhabalia commented on pull request #8555:
URL: https://github.com/apache/pulsar/pull/8555#issuecomment-726950554


   /pulsarbot run-failure-checks



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] rdhabalia commented on pull request #8555: [pulsar-broker] support v1 topic dedup admin api

2020-11-13 Thread GitBox



rdhabalia commented on pull request #8555:
URL: https://github.com/apache/pulsar/pull/8555#issuecomment-726861190


   /pulsarbot run-failure-checks



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] massakam commented on a change in pull request #8561: [broker] Close topics that remain fenced forcefully

2020-11-13 Thread GitBox



massakam commented on a change in pull request #8561:
URL: https://github.com/apache/pulsar/pull/8561#discussion_r523021784



##
File path: conf/broker.conf
##
@@ -452,6 +452,10 @@ systemTopicEnabled=false
 # Please enable the system topic first.
 topicLevelPoliciesEnabled=false
 
+# If a topic remains fenced for this number of seconds, it will be closed 
forcefully.
+# If it is 0 or less, the fenced topic will not be closed.

Review comment:
   Fixed.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] nhathatrinh commented on issue #6834: Apache pulsar Encryption failed using ecdsa

2020-11-13 Thread GitBox



nhathatrinh commented on issue #6834:
URL: https://github.com/apache/pulsar/issues/6834#issuecomment-726819451


   @jiazhai @sijie 
   I am having a same issue with pulsar client 2.6.2. Could you please let me 
know if i miss any dependency? 
   dependency:
   `
org.apache.pulsar
pulsar-client
2.6.2
`
   
   Error:
   `org.apache.pulsar.client.api.PulsarClientException: 
java.lang.NoSuchMethodError: 
org.apache.pulsar.client.api.MessageCrypto.encrypt(Ljava/util/Set;Lorg/apache/pulsar/client/api/CryptoKeyReader;Ljava/util/function/Supplier;Lorg/apache/pulsar/shade/io/netty/buffer/ByteBuf;)Lorg/apache/pulsar/shade/io/netty/buffer/ByteBuf;`



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] Huanli-Meng commented on pull request #8552: Support topic-level DeduplicationSnapshotInterval

2020-11-13 Thread GitBox



Huanli-Meng commented on pull request #8552:
URL: https://github.com/apache/pulsar/pull/8552#issuecomment-726808925


   Pulsar Admin API Doc should also be updated. So i add a doc-required label.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] Huanli-Meng commented on pull request #8561: [broker] Close topics that remain fenced forcefully

2020-11-13 Thread GitBox



Huanli-Meng commented on pull request #8561:
URL: https://github.com/apache/pulsar/pull/8561#issuecomment-726806678


   Add a doc-required label as the broker.config and standalone.config file are 
updated.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[pulsar] branch master updated (6867262 -> 99e9543)

2020-11-13 Thread penghui

This is an automated email from the ASF dual-hosted git repository.

penghui pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git.


from 6867262  Expose consumer names after the mark delete position for the 
Key_Shared subscription (#8545)
 add 99e9543  [Tests] Avoid use Thread.sleep() in TopicPoliciesTest (#8557)

No new revisions were added by this update.

Summary of changes:
 .../broker/service/BrokerServiceException.java |   2 +
 .../pulsar/broker/admin/TopicPoliciesTest.java | 793 ++---
 2 files changed, 373 insertions(+), 422 deletions(-)

[GitHub] [pulsar] Huanli-Meng commented on a change in pull request #8561: [broker] Close topics that remain fenced forcefully

2020-11-13 Thread GitBox



Huanli-Meng commented on a change in pull request #8561:
URL: https://github.com/apache/pulsar/pull/8561#discussion_r522996374



##
File path: conf/broker.conf
##
@@ -452,6 +452,10 @@ systemTopicEnabled=false
 # Please enable the system topic first.
 topicLevelPoliciesEnabled=false
 
+# If a topic remains fenced for this number of seconds, it will be closed 
forcefully.
+# If it is 0 or less, the fenced topic will not be closed.

Review comment:
   ```suggestion
   # If it is set to 0 or a negative number, the fenced topic will not be 
closed.
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[pulsar] branch master updated (6867262 -> 99e9543)

2020-11-13 Thread penghui

This is an automated email from the ASF dual-hosted git repository.

penghui pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git.


from 6867262  Expose consumer names after the mark delete position for the 
Key_Shared subscription (#8545)
 add 99e9543  [Tests] Avoid use Thread.sleep() in TopicPoliciesTest (#8557)

No new revisions were added by this update.

Summary of changes:
 .../broker/service/BrokerServiceException.java |   2 +
 .../pulsar/broker/admin/TopicPoliciesTest.java | 793 ++---
 2 files changed, 373 insertions(+), 422 deletions(-)

[pulsar] branch master updated (6867262 -> 99e9543)

2020-11-13 Thread penghui

This is an automated email from the ASF dual-hosted git repository.

penghui pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git.


from 6867262  Expose consumer names after the mark delete position for the 
Key_Shared subscription (#8545)
 add 99e9543  [Tests] Avoid use Thread.sleep() in TopicPoliciesTest (#8557)

No new revisions were added by this update.

Summary of changes:
 .../broker/service/BrokerServiceException.java |   2 +
 .../pulsar/broker/admin/TopicPoliciesTest.java | 793 ++---
 2 files changed, 373 insertions(+), 422 deletions(-)

[pulsar] branch master updated (6867262 -> 99e9543)

2020-11-13 Thread penghui

This is an automated email from the ASF dual-hosted git repository.

penghui pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git.


from 6867262  Expose consumer names after the mark delete position for the 
Key_Shared subscription (#8545)
 add 99e9543  [Tests] Avoid use Thread.sleep() in TopicPoliciesTest (#8557)

No new revisions were added by this update.

Summary of changes:
 .../broker/service/BrokerServiceException.java |   2 +
 .../pulsar/broker/admin/TopicPoliciesTest.java | 793 ++---
 2 files changed, 373 insertions(+), 422 deletions(-)

[pulsar] branch master updated (6867262 -> 99e9543)

2020-11-13 Thread penghui

This is an automated email from the ASF dual-hosted git repository.

penghui pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git.


from 6867262  Expose consumer names after the mark delete position for the 
Key_Shared subscription (#8545)
 add 99e9543  [Tests] Avoid use Thread.sleep() in TopicPoliciesTest (#8557)

No new revisions were added by this update.

Summary of changes:
 .../broker/service/BrokerServiceException.java |   2 +
 .../pulsar/broker/admin/TopicPoliciesTest.java | 793 ++---
 2 files changed, 373 insertions(+), 422 deletions(-)

[GitHub] [pulsar] codelipenghui merged pull request #8557: [Tests] Avoid use Thread.sleep() in TopicPoliciesTest

2020-11-13 Thread GitBox



codelipenghui merged pull request #8557:
URL: https://github.com/apache/pulsar/pull/8557


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] gaoran10 commented on a change in pull request #8563: [Transaction] Guarantee transaction metadata handlers connect

2020-11-13 Thread GitBox



gaoran10 commented on a change in pull request #8563:
URL: https://github.com/apache/pulsar/pull/8563#discussion_r522951764



##
File path: 
pulsar-client/src/main/java/org/apache/pulsar/client/impl/transaction/TransactionCoordinatorClientImpl.java
##
@@ -75,34 +76,48 @@ public void start() throws 
TransactionCoordinatorClientException {
 public CompletableFuture startAsync() {
 if (STATE_UPDATER.compareAndSet(this, State.NONE, State.STARTING)) {
 return 
pulsarClient.getLookup().getPartitionedTopicMetadata(TopicName.TRANSACTION_COORDINATOR_ASSIGN)
-.thenAccept(partitionMeta -> {
+.thenCompose(partitionMeta -> {
+List> connectFutureList = new 
ArrayList<>();
 if (LOG.isDebugEnabled()) {
 LOG.debug("Transaction meta store assign partition is 
{}.", partitionMeta.partitions);
 }
 if (partitionMeta.partitions > 0) {
 handlers = new 
TransactionMetaStoreHandler[partitionMeta.partitions];
 for (int i = 0; i < partitionMeta.partitions; i++) {
-TransactionMetaStoreHandler handler = new 
TransactionMetaStoreHandler(i, pulsarClient,
-
TopicName.TRANSACTION_COORDINATOR_ASSIGN.toString() + 
TopicName.PARTITIONED_TOPIC_SUFFIX + i);
+CompletableFuture connectFuture = new 
CompletableFuture<>();
+connectFutureList.add(connectFuture);
+TransactionMetaStoreHandler handler = new 
TransactionMetaStoreHandler(
+i, pulsarClient, getTCAssignTopicName(i), 
connectFuture);
 handlers[i] = handler;
 handlerMap.put(i, handler);
 }
 } else {
 handlers = new TransactionMetaStoreHandler[1];
+CompletableFuture connectFuture = new 
CompletableFuture<>();
+connectFutureList.add(connectFuture);
 TransactionMetaStoreHandler handler = new 
TransactionMetaStoreHandler(0, pulsarClient,
-
TopicName.TRANSACTION_COORDINATOR_ASSIGN.toString());
+getTCAssignTopicName(-1), connectFuture);
 handlers[0] = handler;
 handlerMap.put(0, handler);
 }
 
 STATE_UPDATER.set(TransactionCoordinatorClientImpl.this, 
State.READY);
 
+return FutureUtil.waitForAll(connectFutureList);
 });
 } else {
 return FutureUtil.failedFuture(new 
CoordinatorClientStateException("Can not start while current state is " + 
state));
 }
 }
 
+private String getTCAssignTopicName(int partition) {
+if (partition > 0) {

Review comment:
   Ok, I'll fix this.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] codelipenghui commented on a change in pull request #8563: [Transaction] Guarantee transaction metadata handlers connect

2020-11-13 Thread GitBox



codelipenghui commented on a change in pull request #8563:
URL: https://github.com/apache/pulsar/pull/8563#discussion_r522950492



##
File path: 
pulsar-client/src/main/java/org/apache/pulsar/client/impl/transaction/TransactionCoordinatorClientImpl.java
##
@@ -75,34 +76,48 @@ public void start() throws 
TransactionCoordinatorClientException {
 public CompletableFuture startAsync() {
 if (STATE_UPDATER.compareAndSet(this, State.NONE, State.STARTING)) {
 return 
pulsarClient.getLookup().getPartitionedTopicMetadata(TopicName.TRANSACTION_COORDINATOR_ASSIGN)
-.thenAccept(partitionMeta -> {
+.thenCompose(partitionMeta -> {
+List> connectFutureList = new 
ArrayList<>();
 if (LOG.isDebugEnabled()) {
 LOG.debug("Transaction meta store assign partition is 
{}.", partitionMeta.partitions);
 }
 if (partitionMeta.partitions > 0) {
 handlers = new 
TransactionMetaStoreHandler[partitionMeta.partitions];
 for (int i = 0; i < partitionMeta.partitions; i++) {
-TransactionMetaStoreHandler handler = new 
TransactionMetaStoreHandler(i, pulsarClient,
-
TopicName.TRANSACTION_COORDINATOR_ASSIGN.toString() + 
TopicName.PARTITIONED_TOPIC_SUFFIX + i);
+CompletableFuture connectFuture = new 
CompletableFuture<>();
+connectFutureList.add(connectFuture);
+TransactionMetaStoreHandler handler = new 
TransactionMetaStoreHandler(
+i, pulsarClient, getTCAssignTopicName(i), 
connectFuture);
 handlers[i] = handler;
 handlerMap.put(i, handler);
 }
 } else {
 handlers = new TransactionMetaStoreHandler[1];
+CompletableFuture connectFuture = new 
CompletableFuture<>();
+connectFutureList.add(connectFuture);
 TransactionMetaStoreHandler handler = new 
TransactionMetaStoreHandler(0, pulsarClient,
-
TopicName.TRANSACTION_COORDINATOR_ASSIGN.toString());
+getTCAssignTopicName(-1), connectFuture);
 handlers[0] = handler;
 handlerMap.put(0, handler);
 }
 
 STATE_UPDATER.set(TransactionCoordinatorClientImpl.this, 
State.READY);
 
+return FutureUtil.waitForAll(connectFutureList);
 });
 } else {
 return FutureUtil.failedFuture(new 
CoordinatorClientStateException("Can not start while current state is " + 
state));
 }
 }
 
+private String getTCAssignTopicName(int partition) {
+if (partition > 0) {

Review comment:
   ```suggestion
   if (partition >= 0) {
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] Renkai commented on pull request #8556: Reorder imports in some packages to conform checkstyle

2020-11-13 Thread GitBox



Renkai commented on pull request #8556:
URL: https://github.com/apache/pulsar/pull/8556#issuecomment-726761750


   /pulsarbot run-failure-checks



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] BewareMyPower commented on pull request #8554: [C++] Fixed the setting method of linker option.

2020-11-13 Thread GitBox



BewareMyPower commented on pull request #8554:
URL: https://github.com/apache/pulsar/pull/8554#issuecomment-726752433


   /pulsarbot run-failure-checks



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] gaoran10 opened a new pull request #8563: [Transaction] Guarantee transaction metadata handlers connect

2020-11-13 Thread GitBox



gaoran10 opened a new pull request #8563:
URL: https://github.com/apache/pulsar/pull/8563


   ### Motivation
   
   Currently, the transaction metadata handlers start with pulsar client start, 
but the handlers connect with the broker asynchronously, if the client restart, 
the metadata handler may not be available.
   
   ### Modifications
   
   Add the connection future for the metadata handler.
   
   ### Verifying this change
   
   This change added tests and can be verified as follows:
   
 - 
*org.apache.pulsar.client.impl.TransactionEndToEndTest#txnMetadataHandlerRecoverTest*
   
   ### Does this pull request potentially affect one of the following parts:
   
   *If `yes` was chosen, please highlight the changes*
   
 - Dependencies (does it add or upgrade a dependency): (no)
 - The public API: (no)
 - The schema: (no)
 - The default values of configurations: (no)
 - The wire protocol: (no)
 - The rest endpoints: (no)
 - The admin cli options: (no)
 - Anything that affects deployment: (no)
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] codelipenghui commented on a change in pull request #8552: Support topic-level DeduplicationSnapshotInterval

2020-11-13 Thread GitBox



codelipenghui commented on a change in pull request #8552:
URL: https://github.com/apache/pulsar/pull/8552#discussion_r522927597



##
File path: 
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/persistent/TopicDuplicationTest.java
##
@@ -92,6 +92,98 @@ public void testDuplicationApi() throws Exception {
 assertNull(admin.topics().getDeduplicationEnabled(topicName));
 }
 
+@Test(timeOut = 1)
+public void testDuplicationSnapshotApi() throws Exception {
+final String topicName = testTopic + UUID.randomUUID().toString();
+admin.topics().createPartitionedTopic(topicName, 3);
+waitCacheInit(topicName);
+Integer interval = 
admin.topics().getDeduplicationSnapshotInterval(topicName);
+assertNull(interval);
+
+admin.topics().setDeduplicationSnapshotInterval(topicName, 1024);
+for (int i = 0; i < 50; i++) {
+if (admin.topics().getDeduplicationSnapshotInterval(topicName) != 
null) {
+break;
+}
+Thread.sleep(100);
+}
+
Assert.assertEquals(admin.topics().getDeduplicationSnapshotInterval(topicName).intValue(),
 1024);
+admin.topics().removeDeduplicationSnapshotInterval(topicName);
+for (int i = 0; i < 50; i++) {
+if (admin.topics().getDeduplicationSnapshotInterval(topicName) == 
null) {
+break;
+}
+Thread.sleep(100);
+}
+assertNull(admin.topics().getDeduplicationSnapshotInterval(topicName));
+}
+
+@Test(timeOut = 3)
+private void testTopicPolicyTakeSnapshot() throws Exception {
+resetConfig();
+conf.setSystemTopicEnabled(true);
+conf.setTopicLevelPoliciesEnabled(true);
+conf.setBrokerDeduplicationEnabled(true);
+conf.setBrokerDeduplicationSnapshotFrequencyInSeconds(1);
+conf.setBrokerDeduplicationSnapshotIntervalSeconds(4);
+conf.setBrokerDeduplicationEntriesInterval(2);
+super.internalCleanup();
+super.internalSetup();
+super.producerBaseSetup();
+
+final String topicName = testTopic + UUID.randomUUID().toString();
+final String producerName = "my-producer";
+@Cleanup
+Producer producer = pulsarClient
+
.newProducer(Schema.STRING).topic(topicName).enableBatching(false).producerName(producerName).create();
+waitCacheInit(topicName);
+admin.topics().setDeduplicationSnapshotInterval(topicName, 1);
+admin.namespaces().setDeduplicationSnapshotInterval(myNamespace, 2);
+
+int msgNum = 50;
+CountDownLatch countDownLatch = new CountDownLatch(msgNum);
+for (int i = 0; i < msgNum; i++) {
+producer.newMessage().value("msg" + 
i).sendAsync().whenComplete((res, e) -> countDownLatch.countDown());
+}
+countDownLatch.await();
+PersistentTopic persistentTopic = (PersistentTopic) 
pulsar.getBrokerService().getTopicIfExists(topicName).get().get();
+long seqId = 
persistentTopic.getMessageDeduplication().highestSequencedPersisted.get(producerName);
+PositionImpl position = (PositionImpl) 
persistentTopic.getMessageDeduplication().getManagedCursor()
+.getManagedLedger().getLastConfirmedEntry();
+assertEquals(seqId, msgNum - 1);
+assertEquals(position.getEntryId(), msgNum - 1);
+//The first time, use topic-leve policies, 1 second delay + 1 second 
interval
+Thread.sleep(2000);

Review comment:
   I have added awaitility dependency so that we can avoid use sleep in the 
test, for more details you can see https://github.com/apache/pulsar/pull/8557
   





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] codelipenghui commented on pull request #8555: [pulsar-broker] support v1 topic dedup admin api

2020-11-13 Thread GitBox



codelipenghui commented on pull request #8555:
URL: https://github.com/apache/pulsar/pull/8555#issuecomment-726743454


   /pulsarbot run-failure-checks



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] codelipenghui commented on pull request #8554: [C++] Fixed the setting method of linker option.

2020-11-13 Thread GitBox



codelipenghui commented on pull request #8554:
URL: https://github.com/apache/pulsar/pull/8554#issuecomment-726742900


   @BewareMyPower Please help review this PR.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] codelipenghui commented on issue #8551: Is auto delete inactive topic works well on partitioned topic

2020-11-13 Thread GitBox



codelipenghui commented on issue #8551:
URL: https://github.com/apache/pulsar/issues/8551#issuecomment-726741700


   @Shoothzj Looks the same as #8441 and it fixed by #8442



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] Renkai commented on pull request #8556: Reorder imports in some packages to conform checkstyle

2020-11-13 Thread GitBox



Renkai commented on pull request #8556:
URL: https://github.com/apache/pulsar/pull/8556#issuecomment-726736883


   /pulsarbot run-failure-checks



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[pulsar] branch branch-2.6 updated: Expose consumer names after the mark delete position for the Key_Shared subscription (#8545)

2020-11-13 Thread penghui

This is an automated email from the ASF dual-hosted git repository.

penghui pushed a commit to branch branch-2.6
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/branch-2.6 by this push:
 new 7b55504  Expose consumer names after the mark delete position for the 
Key_Shared subscription (#8545)
7b55504 is described below

commit 7b5550476ac052e11cdef13df0b7bcad20c5dd32
Author: lipenghui 
AuthorDate: Fri Nov 13 19:36:01 2020 +0800

Expose consumer names after the mark delete position for the Key_Shared 
subscription (#8545)

1. Expose consumer names after the mark delete position for the Key_Shared 
subscription.
2. Remove the consumer from the recenlyJoinedConsumer depends on the valid 
next position of the next position. Previously, we use the 
position.nextPosition to decide to remove the consumer from the 
recenlyJoinedConsumer but this will lead to consumers can't be deleted 
property. For example, if ledger rollover and the mark delete position is the 
last position of the old ledger and the max read position is the first position 
of the new ledger, In this situation, we should remove the con [...]

So we should get the valid next position for the mark delete position.

Related to #8499

(cherry picked from commit 686726272a76856bb55e3b741e245d05fe2e0274)
---
 pom.xml|  8 ++
 pulsar-broker/pom.xml  |  6 ++
 ...istentStickyKeyDispatcherMultipleConsumers.java | 24 --
 .../service/persistent/PersistentSubscription.java | 10 +++
 .../broker/service/PersistentQueueE2ETest.java | 19 ++---
 .../pulsar/broker/stats/SubscriptionStatsTest.java | 94 ++
 .../common/policies/data/SubscriptionStats.java|  9 +++
 7 files changed, 153 insertions(+), 17 deletions(-)

diff --git a/pom.xml b/pom.xml
index 9946828..8b7a15b 100644
--- a/pom.xml
+++ b/pom.xml
@@ -231,6 +231,7 @@ flexible messaging model and an intuitive client 
API.
 2.3.1
 1.5.0
 5.2.2
+4.0.2
 
 
 0.6.1
@@ -269,6 +270,13 @@ flexible messaging model and an intuitive client 
API.
   
 
   
+org.awaitility
+awaitility
+${awaitility.version}
+test
+  
+
+  
 org.mockito
 mockito-core
 ${mockito.version}
diff --git a/pulsar-broker/pom.xml b/pulsar-broker/pom.xml
index 80b5c49..9b32711 100644
--- a/pulsar-broker/pom.xml
+++ b/pulsar-broker/pom.xml
@@ -170,6 +170,12 @@
   test
 
 
+
+  org.awaitility
+  awaitility
+  test
+
+
 
 
 
diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java
index 1f3d9f5..c2392b7 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java
@@ -34,6 +34,7 @@ import java.util.concurrent.atomic.AtomicInteger;
 import org.apache.bookkeeper.mledger.Entry;
 import org.apache.bookkeeper.mledger.ManagedCursor;
 import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
 import org.apache.bookkeeper.mledger.impl.PositionImpl;
 import org.apache.pulsar.broker.ServiceConfiguration;
 import org.apache.pulsar.broker.service.BrokerServiceException;
@@ -324,15 +325,18 @@ public class 
PersistentStickyKeyDispatcherMultipleConsumers extends PersistentDi
 
 private boolean removeConsumersFromRecentJoinedConsumers() {
 Iterator> itr = 
recentlyJoinedConsumers.entrySet().iterator();
-PositionImpl mdp = (PositionImpl) cursor.getMarkDeletedPosition();
 boolean hasConsumerRemovedFromTheRecentJoinedConsumers = false;
-while (itr.hasNext()) {
-Map.Entry entry = itr.next();
-if (entry.getValue().compareTo(mdp) <= 0) {
-itr.remove();
-hasConsumerRemovedFromTheRecentJoinedConsumers = true;
-} else {
-break;
+PositionImpl mdp = (PositionImpl) cursor.getMarkDeletedPosition();
+if (mdp != null) {
+PositionImpl nextPositionOfTheMarkDeletePosition = 
((ManagedLedgerImpl)cursor.getManagedLedger()).getNextValidPosition(mdp);
+while (itr.hasNext()) {
+Map.Entry entry = itr.next();
+if 
(entry.getValue().compareTo(nextPositionOfTheMarkDeletePosition) <= 0) {
+itr.remove();
+hasConsumerRemovedFromTheRecentJoinedConsumers = true;
+} else {
+break;
+}
 }
 }
 return hasC

[GitHub] [pulsar] 315157973 commented on pull request #7518: Support partitioned topics in the Reader

2020-11-13 Thread GitBox



315157973 commented on pull request #7518:
URL: https://github.com/apache/pulsar/pull/7518#issuecomment-726723032


   /pulsarbot run-failure-checks



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[pulsar] branch master updated: Expose consumer names after the mark delete position for the Key_Shared subscription (#8545)

2020-11-13 Thread penghui

This is an automated email from the ASF dual-hosted git repository.

penghui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 6867262  Expose consumer names after the mark delete position for the 
Key_Shared subscription (#8545)
6867262 is described below

commit 686726272a76856bb55e3b741e245d05fe2e0274
Author: lipenghui 
AuthorDate: Fri Nov 13 19:36:01 2020 +0800

Expose consumer names after the mark delete position for the Key_Shared 
subscription (#8545)

### Motivation

1. Expose consumer names after the mark delete position for the Key_Shared 
subscription.
2. Remove the consumer from the recenlyJoinedConsumer depends on the valid 
next position of the next position. Previously, we use the 
position.nextPosition to decide to remove the consumer from the 
recenlyJoinedConsumer but this will lead to consumers can't be deleted 
property. For example, if ledger rollover and the mark delete position is the 
last position of the old ledger and the max read position is the first position 
of the new ledger, In this situation, we should remove the con [...]

So we should get the valid next position for the mark delete position.

Related to #8499
---
 pom.xml|  8 ++
 pulsar-broker/pom.xml  |  6 ++
 ...istentStickyKeyDispatcherMultipleConsumers.java | 24 --
 .../service/persistent/PersistentSubscription.java | 10 +++
 .../broker/service/PersistentQueueE2ETest.java | 19 ++---
 .../pulsar/broker/stats/SubscriptionStatsTest.java | 94 ++
 .../common/policies/data/SubscriptionStats.java|  9 +++
 7 files changed, 153 insertions(+), 17 deletions(-)

diff --git a/pom.xml b/pom.xml
index 6e0faaf..7ef22c8 100644
--- a/pom.xml
+++ b/pom.xml
@@ -193,6 +193,7 @@ flexible messaging model and an intuitive client 
API.
 1.5.0
 5.2.2
 3.1
+4.0.2
 
 
 0.6.1
@@ -248,6 +249,13 @@ flexible messaging model and an intuitive client 
API.
   
 
   
+org.awaitility
+awaitility
+${awaitility.version}
+test
+  
+
+  
 org.mockito
 mockito-core
 ${mockito.version}
diff --git a/pulsar-broker/pom.xml b/pulsar-broker/pom.xml
index 635143d..7aa742b 100644
--- a/pulsar-broker/pom.xml
+++ b/pulsar-broker/pom.xml
@@ -169,6 +169,12 @@
   test
 
 
+
+  org.awaitility
+  awaitility
+  test
+
+
 
 
 
diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java
index 3658567..56963e8 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java
@@ -34,6 +34,7 @@ import java.util.concurrent.atomic.AtomicInteger;
 import org.apache.bookkeeper.mledger.Entry;
 import org.apache.bookkeeper.mledger.ManagedCursor;
 import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
 import org.apache.bookkeeper.mledger.impl.PositionImpl;
 import org.apache.pulsar.broker.ServiceConfiguration;
 import org.apache.pulsar.broker.service.BrokerServiceException;
@@ -325,15 +326,18 @@ public class 
PersistentStickyKeyDispatcherMultipleConsumers extends PersistentDi
 
 private boolean removeConsumersFromRecentJoinedConsumers() {
 Iterator> itr = 
recentlyJoinedConsumers.entrySet().iterator();
-PositionImpl mdp = (PositionImpl) cursor.getMarkDeletedPosition();
 boolean hasConsumerRemovedFromTheRecentJoinedConsumers = false;
-while (itr.hasNext()) {
-Map.Entry entry = itr.next();
-if (entry.getValue().compareTo(mdp) <= 0) {
-itr.remove();
-hasConsumerRemovedFromTheRecentJoinedConsumers = true;
-} else {
-break;
+PositionImpl mdp = (PositionImpl) cursor.getMarkDeletedPosition();
+if (mdp != null) {
+PositionImpl nextPositionOfTheMarkDeletePosition = 
((ManagedLedgerImpl)cursor.getManagedLedger()).getNextValidPosition(mdp);
+while (itr.hasNext()) {
+Map.Entry entry = itr.next();
+if 
(entry.getValue().compareTo(nextPositionOfTheMarkDeletePosition) <= 0) {
+itr.remove();
+hasConsumerRemovedFromTheRecentJoinedConsumers = true;
+} else {
+break;
+}
 }
 }
 return hasConsumerRemovedFromTheRecentJoinedConsumers;
@@ -361,6 +365,10 @@

[GitHub] [pulsar] codelipenghui merged pull request #8545: Expose consumer names after the mark delete position for the Key_Shared subscription

2020-11-13 Thread GitBox



codelipenghui merged pull request #8545:
URL: https://github.com/apache/pulsar/pull/8545


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] congbobo184 commented on pull request #8541: [Transaction] Fix the Github CI Test

2020-11-13 Thread GitBox



congbobo184 commented on pull request #8541:
URL: https://github.com/apache/pulsar/pull/8541#issuecomment-726713111


   /pulsarbot run-failure-checks



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] hangc0276 commented on pull request #8557: [Tests] Avoid use Thread.sleep() in TopicPoliciesTest

2020-11-13 Thread GitBox



hangc0276 commented on pull request #8557:
URL: https://github.com/apache/pulsar/pull/8557#issuecomment-726712000


   /pulsarbot run-failure-checks



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] aahmed-se commented on issue #8562: [Question] Last-value-queues support

2020-11-13 Thread GitBox



aahmed-se commented on issue #8562:
URL: https://github.com/apache/pulsar/issues/8562#issuecomment-726707973


   pulsar supports topic compaction so in essence it provides the same 
functionality



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] strangeman375 opened a new issue #8562: [Question] Last-value-queues support

2020-11-13 Thread GitBox



strangeman375 opened a new issue #8562:
URL: https://github.com/apache/pulsar/issues/8562


   Hello!
   Does Pulsar support last-value-queue, like 
[artemis](https://activemq.apache.org/components/artemis/documentation/latest/last-value-queues.html)?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] Renkai commented on pull request #8556: Reorder imports in some packages to conform checkstyle

2020-11-13 Thread GitBox



Renkai commented on pull request #8556:
URL: https://github.com/apache/pulsar/pull/8556#issuecomment-726696587


   /pulsarbot run-failure-checks



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] massakam opened a new pull request #8561: [broker] Close topics that remain fenced forcefully

2020-11-13 Thread GitBox



massakam opened a new pull request #8561:
URL: https://github.com/apache/pulsar/pull/8561


   ### Motivation
   
   The other day, we faced a problem where a topic remained fenced and 
unavailable. This topic remained unavailable until it was unloaded. The 
following is the broker log at that time.
   ```
   11:37:55.905 [bookkeeper-ml-workers-OrderedExecutor-77-0] INFO  
o.a.b.mledger.impl.OpAddEntry- [tenant/ns/persistent/topic] Closing 
ledger 40891546 for being full
   11:37:56.208 [pulsar-ordered-OrderedExecutor-0-0-EventThread] ERROR 
o.a.b.client.MetadataUpdateLoop  - 
UpdateLoop(ledgerId=40891546,loopId=6ce63876) Error writing metadata to store
   11:37:56.209 [pulsar-ordered-OrderedExecutor-0-0-EventThread] WARN  
o.a.b.mledger.impl.OpAddEntry- Error when closing ledger 40891546. 
Status=Error while using ZooKeeper
   11:37:56.359 [pulsar-ordered-OrderedExecutor-0-0-EventThread] ERROR 
o.a.b.mledger.impl.ManagedLedgerImpl - [tenant/ns/persistent/topic] Error 
creating ledger rc=-9 Error while using ZooKeeper
   11:37:56.359 [pulsar-ordered-OrderedExecutor-0-0-EventThread] INFO  
o.a.pulsar.broker.service.Producer   - Disconnecting producer: 
Producer{topic=PersistentTopic{topic=persistent://tenant/ns/topic}, 
client=/xxx.xxx.xxx.xxx:40646, producerName=pulsar.repl.jp-west, producerId=668}
   11:37:56.360 [pulsar-ordered-OrderedExecutor-0-0-EventThread] WARN  
o.a.p.b.s.persistent.PersistentTopic - [persistent://tenant/ns/topic] Failed to 
persist msg in store: Error while using ZooKeeper
   11:37:56.360 [pulsar-ordered-OrderedExecutor-0-0-EventThread] INFO  
o.a.pulsar.broker.service.Producer   - Disconnecting producer: 
Producer{topic=PersistentTopic{topic=persistent://tenant/ns/topic}, 
client=/xxx.xxx.xxx.xxx:40646, producerName=pulsar.repl.jp-west, producerId=668}
   11:37:56.360 [pulsar-ordered-OrderedExecutor-0-0-EventThread] WARN  
o.a.p.b.s.persistent.PersistentTopic - [persistent://tenant/ns/topic] Failed to 
persist msg in store: Error while using ZooKeeper
   11:37:56.360 [pulsar-ordered-OrderedExecutor-0-0-EventThread] INFO  
o.a.pulsar.broker.service.Producer   - Disconnecting producer: 
Producer{topic=PersistentTopic{topic=persistent://tenant/ns/topic}, 
client=/xxx.xxx.xxx.xxx:40646, producerName=pulsar.repl.jp-west, producerId=668}
   11:37:56.360 [pulsar-ordered-OrderedExecutor-0-0-EventThread] WARN  
o.a.p.b.s.persistent.PersistentTopic - [persistent://tenant/ns/topic] Failed to 
persist msg in store: Error while using ZooKeeper
   11:37:56.360 [pulsar-ordered-OrderedExecutor-0-0-EventThread] WARN  
o.a.p.b.s.persistent.PersistentTopic - [persistent://tenant/ns/topic] Failed to 
persist msg in store: Error while using ZooKeeper
   11:37:56.360 [pulsar-ordered-OrderedExecutor-0-0-EventThread] WARN  
o.a.p.b.s.persistent.PersistentTopic - [persistent://tenant/ns/topic] Failed to 
persist msg in store: Error while using ZooKeeper
   11:37:56.360 [pulsar-ordered-OrderedExecutor-0-0-EventThread] WARN  
o.a.p.b.s.persistent.PersistentTopic - [persistent://tenant/ns/topic] Failed to 
persist msg in store: Error while using ZooKeeper
   11:37:57.495 [ForkJoinPool.commonPool-worker-51] INFO  
o.a.pulsar.broker.service.ServerCnx  - 
[/xxx.xxx.xxx.xxx:40256][persistent://tenant/ns/topic] Creating producer. 
producerId=668
   11:37:58.291 [bookkeeper-ml-workers-OrderedExecutor-77-0] INFO  
o.a.b.mledger.impl.ManagedLedgerImpl - [tenant/ns/persistent/topic] End 
TrimConsumedLedgers. ledgers=2 totalSize=162868668
   11:37:58.291 [bookkeeper-ml-workers-OrderedExecutor-77-0] INFO  
o.a.b.mledger.impl.ManagedLedgerImpl - [tenant/ns/persistent/topic] Removing 
ledger 40880508 - size: 82183409
   11:37:58.292 [ForkJoinPool.commonPool-worker-20] INFO  
o.a.pulsar.broker.service.ServerCnx  - [/xxx.xxx.xxx.xxx:40256]-668 
persistent://tenant/ns/topic configured with schema false
   11:37:58.292 [ForkJoinPool.commonPool-worker-20] WARN  
o.a.p.b.s.persistent.PersistentTopic - [persistent://tenant/ns/topic] 
Attempting to add producer to a fenced topic
   11:37:58.292 [ForkJoinPool.commonPool-worker-20] ERROR 
o.a.pulsar.broker.service.ServerCnx  - [/xxx.xxx.xxx.xxx:40256] Failed to add 
producer to topic persistent://tenant/ns/topic: Topic is temporarily unavailable
   11:37:58.728 [ForkJoinPool.commonPool-worker-75] INFO  
o.a.pulsar.broker.service.ServerCnx  - 
[/xxx.xxx.xxx.xxx:40330][persistent://tenant/ns/topic] Creating producer. 
producerId=668
   11:37:58.729 [ForkJoinPool.commonPool-worker-75] INFO  
o.a.pulsar.broker.service.ServerCnx  - [/xxx.xxx.xxx.xxx:40330]-668 
persistent://tenant/ns/topic configured with schema false
   11:37:58.729 [ForkJoinPool.commonPool-worker-75] WARN  
o.a.p.b.s.persistent.PersistentTopic - [persistent://tenant/ns/topic] 
Attempting to add producer to a fenced topic
   11:37:58.729 [ForkJoinPool.commonPool-worker-75] ERROR 
o.a.pulsar.broker.service.ServerCnx  - [/xxx.xxx.xxx.xxx:40330] Failed to add 
producer to topic persistent://tenant/n

[GitHub] [pulsar] hnail commented on pull request #8372: [schemaregistry]ProtobufNative Schema Support

2020-11-13 Thread GitBox



hnail commented on pull request #8372:
URL: https://github.com/apache/pulsar/pull/8372#issuecomment-72439


   /pulsarbot run-failure-checks



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-client-go] freeznet commented on issue #172: Support key based batcher for key_shared

2020-11-13 Thread GitBox



freeznet commented on issue #172:
URL: 
https://github.com/apache/pulsar-client-go/issues/172#issuecomment-726663132


   I can try to implement this feature 🔢 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] andrekramer1 commented on pull request #8544: Consumer filtering

2020-11-13 Thread GitBox



andrekramer1 commented on pull request #8544:
URL: https://github.com/apache/pulsar/pull/8544#issuecomment-726659467


   Hi,
   
   The performance impact when not filtering should be minimal but there is 
copying and repackaging of messages happening when filtering message batches on 
the broker. We did some performance measurements to show that there is a 
significant saving when *most* messages are filtered out.
   
   Andre
   
   From: joefk 
   Sent: 12 November 2020 18:26
   To: apache/pulsar 
   Cc: Kramer, Andre ; Author 

   Subject: Re: [apache/pulsar] Consumer filtering (#8544)
   
   
   What is the performance impact on broker througput with this feature? Has 
any tests been run for measuring this?
   What additional GC load is generated by this feature?
   
   —
   You are receiving this because you authored the thread.
   Reply to this email directly, view it on 
GitHub, or 
unsubscribe.
   
   This communication contains information which is confidential and may also 
be privileged. It is for the exclusive use of the intended recipient(s). If you 
are not the intended recipient(s), please note that any distribution, copying, 
or use of this communication or the information in it, is strictly prohibited. 
If you have received this communication in error please notify us by e-mail and 
then delete the e-mail and any copies of it.
   Software AG (UK) Limited Registered in England & Wales 1310740 - 
http://www.softwareag.com/uk
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] cbornet commented on a change in pull request #6720: Make ServerCnx, Producer and Consumer independent of Netty

2020-11-13 Thread GitBox



cbornet commented on a change in pull request #6720:
URL: https://github.com/apache/pulsar/pull/6720#discussion_r522822050



##
File path: 
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PulsarCommandSender.java
##
@@ -26,39 +29,92 @@
 
 public interface PulsarCommandSender {
 
-
-void sendPartitionMetadataResponse(PulsarApi.ServerError error, String 
errorMsg, long requestId);
-
-void sendPartitionMetadataResponse(int partitions, long requestId);
-
-void sendSuccessResponse(long requestId);
-
-void sendErrorResponse(long requestId, PulsarApi.ServerError error, String 
message);
-
-void sendProducerSuccessResponse(long requestId, String producerName, 
SchemaVersion schemaVersion);
-
-void sendProducerSuccessResponse(long requestId, String producerName, long 
lastSequenceId,
- SchemaVersion schemaVersion);
-
-void sendSendReceiptResponse(long producerId, long sequenceId, long 
highestId, long ledgerId,
- long entryId);
-
-void sendSendError(long producerId, long sequenceId, PulsarApi.ServerError 
error, String errorMsg);
-
-void sendGetTopicsOfNamespaceResponse(List topics, long requestId);
-
-void sendGetSchemaResponse(long requestId, SchemaInfo schema, 
SchemaVersion version);
-
-void sendGetSchemaErrorResponse(long requestId, PulsarApi.ServerError 
error, String errorMessage);
-
-void sendGetOrCreateSchemaResponse(long requestId, SchemaVersion 
schemaVersion);
-
-void sendGetOrCreateSchemaErrorResponse(long requestId, 
PulsarApi.ServerError error, String errorMessage);
-
-void sendConnectedResponse(int clientProtocolVersion, int maxMessageSize);
-
-void sendLookupResponse(String brokerServiceUrl, String 
brokerServiceUrlTls, boolean authoritative,
-PulsarApi.CommandLookupTopicResponse.LookupType 
response, long requestId, boolean proxyThroughServiceUrl);
-
-void sendLookupResponse(PulsarApi.ServerError error, String errorMsg, long 
requestId);
+default void sendPartitionMetadataResponse(PulsarApi.ServerError error, 
String errorMsg, long requestId) {

Review comment:
   I've set it back to non-default. We can see about the interest of 
default later.

##
File path: 
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/TransportCnx.java
##
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service;
+
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.pulsar.broker.authentication.AuthenticationDataSource;
+import org.apache.pulsar.common.api.proto.PulsarApi.ServerError;
+
+import java.net.SocketAddress;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+
+public interface TransportCnx {

Review comment:
   I've set it back to non-default. We can see about the interest of 
default later.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] Renkai commented on pull request #8556: Reorder imports in some packages to conform checkstyle

2020-11-13 Thread GitBox



Renkai commented on pull request #8556:
URL: https://github.com/apache/pulsar/pull/8556#issuecomment-726644394


   /pulsarbot run-failure-checks



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] 315157973 commented on pull request #8552: Support topic-level DeduplicationSnapshotInterval

2020-11-13 Thread GitBox



315157973 commented on pull request #8552:
URL: https://github.com/apache/pulsar/pull/8552#issuecomment-726640146


   /pulsarbot run-failure-checks



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] hnail commented on a change in pull request #8372: [schemaregistry]ProtobufNative Schema Support

2020-11-13 Thread GitBox



hnail commented on a change in pull request #8372:
URL: https://github.com/apache/pulsar/pull/8372#discussion_r522814157



##
File path: 
pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/ProtobufNativeSchemaUtils.java
##
@@ -0,0 +1,139 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl.schema;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.pulsar.client.api.SchemaSerializationException;
+import org.apache.pulsar.common.protocol.schema.ProtobufNativeSchemaData;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static com.google.protobuf.DescriptorProtos.FileDescriptorProto;
+import static com.google.protobuf.DescriptorProtos.FileDescriptorSet;
+import static com.google.protobuf.Descriptors.*;

Review comment:
   @codelipenghui thanks for review , fixed as Suggestion。





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] jiazhai commented on issue #8553: [Pulsar Broker] support apache ranger

2020-11-13 Thread GitBox



jiazhai commented on issue #8553:
URL: https://github.com/apache/pulsar/issues/8553#issuecomment-726599622


   Thanks @ltamber for this initiative. It would be a great supplementary to 
current authorization. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] sijie opened a new pull request #8560: [Functions] Provide an interface for functions worker service

2020-11-13 Thread GitBox



sijie opened a new pull request #8560:
URL: https://github.com/apache/pulsar/pull/8560


   *Motivation*
   
   Make the pulsar functions worker serve as an interface to allow plugin 
different functions of worker service implementations.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[pulsar] branch master updated (3b2c852 -> 67f544c)

2020-11-13 Thread sijie

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git.


from 3b2c852   PIP-72: Introduce Pulsar Interface Taxonomy: Audience and 
Stability Classification  (#8530)
 add 67f544c  support websocket producer for v2 topic (#8535)

No new revisions were added by this update.

Summary of changes:
 .../org/apache/pulsar/proxy/socket/client/PerformanceClient.java| 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)

[GitHub] [pulsar] sijie merged pull request #8535: [pulsar-perf] Support WebSocket Producer for V2 Topics

2020-11-13 Thread GitBox



sijie merged pull request #8535:
URL: https://github.com/apache/pulsar/pull/8535


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

63 matches

Mail list logo