[GitHub] rdhabalia commented on issue #1223: Add a `internalConfiguration` admin restful endpoint for query internal configuration
rdhabalia commented on issue #1223: Add a `internalConfiguration` admin restful endpoint for query internal configuration URL: https://github.com/apache/incubator-pulsar/pull/1223#issuecomment-365839106 LGTM.. just few minor comments.. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] rdhabalia commented on a change in pull request #1223: Add a `internalConfiguration` admin restful endpoint for query internal configuration
rdhabalia commented on a change in pull request #1223: Add a `internalConfiguration` admin restful endpoint for query internal configuration URL: https://github.com/apache/incubator-pulsar/pull/1223#discussion_r168390204 ## File path: pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/PulsarAdmin.java ## @@ -154,6 +156,7 @@ public PulsarAdmin(URL serviceUrl, ClientConfiguration pulsarConfig) throws Puls WebTarget root = client.target(serviceUrl.toString()); web = root.path("/admin"); +this.internalConfiguration = new InternalConfigurationImpl(web, auth); Review comment: should it be part of `brokers` as this configuration has been configured at broker (eg: `pulsar().getConfiguration().getxxx()`? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] rdhabalia commented on a change in pull request #1223: Add a `internalConfiguration` admin restful endpoint for query internal configuration
rdhabalia commented on a change in pull request #1223: Add a `internalConfiguration` admin restful endpoint for query internal configuration URL: https://github.com/apache/incubator-pulsar/pull/1223#discussion_r168390403 ## File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/InternalConfiguration.java ## @@ -0,0 +1,50 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pulsar.broker.admin; + +import io.swagger.annotations.Api; +import io.swagger.annotations.ApiOperation; +import io.swagger.annotations.ApiResponse; +import io.swagger.annotations.ApiResponses; +import javax.ws.rs.GET; +import javax.ws.rs.Path; +import javax.ws.rs.Produces; +import javax.ws.rs.core.MediaType; +import org.apache.bookkeeper.conf.ClientConfiguration; +import org.apache.pulsar.common.conf.InternalConfigurationData; + +@Path("/internal-configuration") +@Api(value = "/internal-configuration", description = "InternalConfiguration admin apis", tags = "internal-configuration") +@Produces(MediaType.APPLICATION_JSON) +public class InternalConfiguration extends AdminResource { + +@GET +@ApiOperation(value = "Get the internal configuration data", response = InternalConfigurationData.class) +@ApiResponses(value = { +@ApiResponse(code = 403, message = "Don't have admin permission") Review comment: in which scenario it returns 403? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat closed issue #1058: Provide a mechanism to tell whether a consumer is the leader of a failover subscription
merlimat closed issue #1058: Provide a mechanism to tell whether a consumer is the leader of a failover subscription URL: https://github.com/apache/incubator-pulsar/issues/1058 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch master updated: Introduce ActiveConsumerListener for realizing if a consumer is active in a failover subscription group (#1156)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git The following commit(s) were added to refs/heads/master by this push: new f2160c0 Introduce ActiveConsumerListener for realizing if a consumer is active in a failover subscription group (#1156) f2160c0 is described below commit f2160c01e3581f0e8374db5d4d713810de7533af Author: Sijie GuoAuthorDate: Thu Feb 15 12:12:00 2018 +0800 Introduce ActiveConsumerListener for realizing if a consumer is active in a failover subscription group (#1156) * Introduce ConsumerGroupListener for realizing if a consumer is active in a failover subscription group * Rename ConsumerGroupListener to ActiveConsumerListener * Fail subscribe if active consumer listener is provided with non-failover subscription. * Notify consumer state change only after the cursor is rewinded. * Address comments * Rename ActiveConsumerListener to ConsumerEventListener Use consumer object as identifier for comparison * Fix test after rebase * Fix license headers * Ignore active consumer change command in cpp client * rename "become" to "became" --- .../AbstractDispatcherSingleActiveConsumer.java| 38 +- .../org/apache/pulsar/broker/service/Consumer.java | 15 + .../PersistentDispatcherSingleActiveConsumer.java | 10 +- .../PersistentDispatcherFailoverConsumerTest.java | 157 ++- .../broker/service/PersistentFailoverE2ETest.java | 96 +++- pulsar-client-cpp/lib/ClientConnection.cc | 7 + pulsar-client-cpp/lib/Commands.cc | 3 + .../pulsar/client/api/ConsumerConfiguration.java | 29 ++ .../pulsar/client/api/ConsumerEventListener.java | 36 ++ .../org/apache/pulsar/client/impl/ClientCnx.java | 14 + .../apache/pulsar/client/impl/ConsumerBase.java| 3 + .../apache/pulsar/client/impl/ConsumerImpl.java| 14 + .../client/impl/PartitionedConsumerImpl.java | 4 +- .../pulsar/client/impl/PulsarClientImpl.java | 7 + .../org/apache/pulsar/common/api/Commands.java | 18 + .../apache/pulsar/common/api/PulsarDecoder.java| 11 +- .../apache/pulsar/common/api/proto/PulsarApi.java | 486 + pulsar-common/src/main/proto/PulsarApi.proto | 12 +- .../pulsar/common/api/PulsarDecoderTest.java | 60 +++ 19 files changed, 993 insertions(+), 27 deletions(-) diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractDispatcherSingleActiveConsumer.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractDispatcherSingleActiveConsumer.java index 5181df1..1b37f89 100644 --- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractDispatcherSingleActiveConsumer.java +++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractDispatcherSingleActiveConsumer.java @@ -26,9 +26,9 @@ import java.util.concurrent.atomic.AtomicIntegerFieldUpdater; import java.util.concurrent.atomic.AtomicReferenceFieldUpdater; import org.apache.pulsar.broker.service.BrokerServiceException; -import org.apache.pulsar.broker.service.Consumer; import org.apache.pulsar.broker.service.BrokerServiceException.ConsumerBusyException; import org.apache.pulsar.broker.service.BrokerServiceException.ServerMetadataException; +import org.apache.pulsar.broker.service.Consumer; import org.apache.pulsar.common.api.proto.PulsarApi.CommandSubscribe.SubType; import org.apache.pulsar.utils.CopyOnWriteArrayList; import org.slf4j.Logger; @@ -72,7 +72,17 @@ public abstract class AbstractDispatcherSingleActiveConsumer { protected abstract boolean isConsumersExceededOnSubscription(); -protected void pickAndScheduleActiveConsumer() { +protected void notifyActiveConsumerChanged(Consumer activeConsumer) { +if (null != activeConsumer && subscriptionType == SubType.Failover) { +consumers.forEach(consumer -> +consumer.notifyActiveConsumerChange(activeConsumer)); +} +} + +/** + * @return the previous active consumer if the consumer is changed, otherwise null. + */ +protected boolean pickAndScheduleActiveConsumer() { checkArgument(!consumers.isEmpty()); consumers.sort((c1, c2) -> c1.consumerName().compareTo(c2.consumerName())); @@ -80,12 +90,15 @@ public abstract class AbstractDispatcherSingleActiveConsumer { int index = partitionIndex % consumers.size(); Consumer prevConsumer = ACTIVE_CONSUMER_UPDATER.getAndSet(this, consumers.get(index)); -if (prevConsumer == ACTIVE_CONSUMER_UPDATER.get(this)) { +Consumer activeConsumer = ACTIVE_CONSUMER_UPDATER.get(this); +if (prevConsumer == activeConsumer) { // Active consumer did not change. Do nothing at this point -
[GitHub] merlimat closed pull request #1156: Introduce ActiveConsumerListener for realizing if a consumer is active in a failover subscription group
merlimat closed pull request #1156: Introduce ActiveConsumerListener for realizing if a consumer is active in a failover subscription group URL: https://github.com/apache/incubator-pulsar/pull/1156 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractDispatcherSingleActiveConsumer.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractDispatcherSingleActiveConsumer.java index 5181df177..1b37f8946 100644 --- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractDispatcherSingleActiveConsumer.java +++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractDispatcherSingleActiveConsumer.java @@ -26,9 +26,9 @@ import java.util.concurrent.atomic.AtomicReferenceFieldUpdater; import org.apache.pulsar.broker.service.BrokerServiceException; -import org.apache.pulsar.broker.service.Consumer; import org.apache.pulsar.broker.service.BrokerServiceException.ConsumerBusyException; import org.apache.pulsar.broker.service.BrokerServiceException.ServerMetadataException; +import org.apache.pulsar.broker.service.Consumer; import org.apache.pulsar.common.api.proto.PulsarApi.CommandSubscribe.SubType; import org.apache.pulsar.utils.CopyOnWriteArrayList; import org.slf4j.Logger; @@ -72,7 +72,17 @@ public AbstractDispatcherSingleActiveConsumer(SubType subscriptionType, int part protected abstract boolean isConsumersExceededOnSubscription(); -protected void pickAndScheduleActiveConsumer() { +protected void notifyActiveConsumerChanged(Consumer activeConsumer) { +if (null != activeConsumer && subscriptionType == SubType.Failover) { +consumers.forEach(consumer -> +consumer.notifyActiveConsumerChange(activeConsumer)); +} +} + +/** + * @return the previous active consumer if the consumer is changed, otherwise null. + */ +protected boolean pickAndScheduleActiveConsumer() { checkArgument(!consumers.isEmpty()); consumers.sort((c1, c2) -> c1.consumerName().compareTo(c2.consumerName())); @@ -80,12 +90,15 @@ protected void pickAndScheduleActiveConsumer() { int index = partitionIndex % consumers.size(); Consumer prevConsumer = ACTIVE_CONSUMER_UPDATER.getAndSet(this, consumers.get(index)); -if (prevConsumer == ACTIVE_CONSUMER_UPDATER.get(this)) { +Consumer activeConsumer = ACTIVE_CONSUMER_UPDATER.get(this); +if (prevConsumer == activeConsumer) { // Active consumer did not change. Do nothing at this point -return; +return false; +} else { +// If the active consumer is changed, send notification. +scheduleReadOnActiveConsumer(); +return true; } - -scheduleReadOnActiveConsumer(); } public synchronized void addConsumer(Consumer consumer) throws BrokerServiceException { @@ -109,8 +122,17 @@ public synchronized void addConsumer(Consumer consumer) throws BrokerServiceExce consumers.add(consumer); -// Pick an active consumer and start it -pickAndScheduleActiveConsumer(); +if (!pickAndScheduleActiveConsumer()) { +// the active consumer is not changed +Consumer currentActiveConsumer = ACTIVE_CONSUMER_UPDATER.get(this); +if (null == currentActiveConsumer) { +if (log.isDebugEnabled()) { +log.debug("Current active consumer disappears while adding consumer {}", consumer); +} +} else { +consumer.notifyActiveConsumerChange(currentActiveConsumer); +} +} } diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java index fd77ef290..0bb30cfb6 100644 --- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java +++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java @@ -152,6 +152,21 @@ public String consumerName() { return consumerName; } +void notifyActiveConsumerChange(Consumer activeConsumer) { +if (!Commands.peerSupportsActiveConsumerListener(cnx.getRemoteEndpointProtocolVersion())) { +// if the client is older than `v12`, we don't need to send consumer group changes. +return; +} + +if (log.isDebugEnabled()) { +log.debug("notify consumer {} - that [{}] for subscription {} has new active consumer : {}", +consumerId, topicName, subscription.getName(), activeConsumer); +} +
svn commit: r25081 - /dev/incubator/pulsar/pulsar-1.22.0-incubating-candidate-0/
Author: jai1 Date: Thu Feb 15 02:06:00 2018 New Revision: 25081 Log: Staging artifacts and signature for Pulsar release 1.22.0-incubating Added: dev/incubator/pulsar/pulsar-1.22.0-incubating-candidate-0/apache-pulsar-1.22.0-incubating-bin.tar.gz (with props) dev/incubator/pulsar/pulsar-1.22.0-incubating-candidate-0/apache-pulsar-1.22.0-incubating-bin.tar.gz.asc dev/incubator/pulsar/pulsar-1.22.0-incubating-candidate-0/apache-pulsar-1.22.0-incubating-bin.tar.gz.md5 dev/incubator/pulsar/pulsar-1.22.0-incubating-candidate-0/apache-pulsar-1.22.0-incubating-bin.tar.gz.sha512 Modified: dev/incubator/pulsar/pulsar-1.22.0-incubating-candidate-0/apache-pulsar-1.22.0-incubating-src.tar.gz.asc Added: dev/incubator/pulsar/pulsar-1.22.0-incubating-candidate-0/apache-pulsar-1.22.0-incubating-bin.tar.gz == Binary file - no diff available. Propchange: dev/incubator/pulsar/pulsar-1.22.0-incubating-candidate-0/apache-pulsar-1.22.0-incubating-bin.tar.gz -- svn:mime-type = application/octet-stream Added: dev/incubator/pulsar/pulsar-1.22.0-incubating-candidate-0/apache-pulsar-1.22.0-incubating-bin.tar.gz.asc == --- dev/incubator/pulsar/pulsar-1.22.0-incubating-candidate-0/apache-pulsar-1.22.0-incubating-bin.tar.gz.asc (added) +++ dev/incubator/pulsar/pulsar-1.22.0-incubating-candidate-0/apache-pulsar-1.22.0-incubating-bin.tar.gz.asc Thu Feb 15 02:06:00 2018 @@ -0,0 +1,16 @@ +-BEGIN PGP SIGNATURE- + +iQIzBAABCAAdFiEEDQCP4t9TLRC/fG0se6GmTLvBFOwFAlqE6tAACgkQe6GmTLvB +FOxOCg/+LOavQKTSrdrtpX6gUCVuXvnDABAAgE37tRNJfsLwmfC8G+ZaNyb6vSLf +nbCnAwNwi/Nj0X/4H9ht4mBxrJZ78Ii3UtBjT3lA4naMDZcyPmN/5ISY5RNscVln +wMi9nt8DdWKV3XR0XWHFx9tgrKOhPfkb9xdTturljj8/a3oW3yhOFpBaUve+KFkJ +CX3ibEAv9Cd5iIvrv0rPTUdPxsK2ht7JYewdLUxtHhAJmWkFKcdT8InLqrqb2tmE +IqT9i/SY2LGW+hyq28pdnjzwa5g6Qu6ilrN69LRBJ/5oSwvfS8+H3C9yPYDh2WVF +vsYclpy1BQftRPqR+LAyIZcmotywmR6CNkGRBEj+/M+CsFpc06E0ixPWlJDV9UEb +adSCnHmohwwD3TS8OmMNI/O7ipcLB5Mamor6Gi9SYjHA+pM0AG+sSRpzV67YxdIi ++x61YBeZ1OnmYhlj9b0tFSMCkQXwlnxoIC1liU8rG133V2+9wt6W4BN4TjB74rWW +50mECdvGDhjty9AF1HbuA54PA2/GwKV2eW0eAd7DYCq1U+CaP7yUrSagSknAnvXB +50tm6EaSt4Hx2S32xvUVg4pn86XAfO6ePOKywCExHQxqDzenTkvMeYssZbnPHxxe +FBYUa0EqSJmbkJ7cq+Hjzv8RNchEjmEWdL1qOquYZeN/qfXXazE= +=/xWX +-END PGP SIGNATURE- Added: dev/incubator/pulsar/pulsar-1.22.0-incubating-candidate-0/apache-pulsar-1.22.0-incubating-bin.tar.gz.md5 == --- dev/incubator/pulsar/pulsar-1.22.0-incubating-candidate-0/apache-pulsar-1.22.0-incubating-bin.tar.gz.md5 (added) +++ dev/incubator/pulsar/pulsar-1.22.0-incubating-candidate-0/apache-pulsar-1.22.0-incubating-bin.tar.gz.md5 Thu Feb 15 02:06:00 2018 @@ -0,0 +1,2 @@ +apache-pulsar-1.22.0-incubating-bin.tar.gz: +98 41 A5 01 81 98 9F 4F 44 E7 0D 6B 1D 15 3E E2 Added: dev/incubator/pulsar/pulsar-1.22.0-incubating-candidate-0/apache-pulsar-1.22.0-incubating-bin.tar.gz.sha512 == --- dev/incubator/pulsar/pulsar-1.22.0-incubating-candidate-0/apache-pulsar-1.22.0-incubating-bin.tar.gz.sha512 (added) +++ dev/incubator/pulsar/pulsar-1.22.0-incubating-candidate-0/apache-pulsar-1.22.0-incubating-bin.tar.gz.sha512 Thu Feb 15 02:06:00 2018 @@ -0,0 +1,3 @@ +apache-pulsar-1.22.0-incubating-bin.tar.gz: +408DDB47 98355343 23626766 EB2FFD64 7E77105C F7F6FE92 AC42502C A2FBC632 6BCECD5C + 43177395 643A10E7 F080B159 ECB4A2A5 D2D01508 83EBED26 6CECDA9A Modified: dev/incubator/pulsar/pulsar-1.22.0-incubating-candidate-0/apache-pulsar-1.22.0-incubating-src.tar.gz.asc == --- dev/incubator/pulsar/pulsar-1.22.0-incubating-candidate-0/apache-pulsar-1.22.0-incubating-src.tar.gz.asc (original) +++ dev/incubator/pulsar/pulsar-1.22.0-incubating-candidate-0/apache-pulsar-1.22.0-incubating-src.tar.gz.asc Thu Feb 15 02:06:00 2018 @@ -1,16 +1,16 @@ -BEGIN PGP SIGNATURE- -iQIzBAABCAAdFiEEDQCP4t9TLRC/fG0se6GmTLvBFOwFAlqCtq8ACgkQe6GmTLvB -FOyEwhAAt0OFdejzQTJi7KA2Qv+mhazVuBEZfygwyRjNy7pNoX+CI0tIDCo6myDd -jfUjx3Fg6hFR3WGzLW0+n+meHxsWrcpiqoR6vV7PjH+CpxJXY6+ZrS3/0QuDdAT/ -/ZlSStC8S3AT6mSRJzjab9HGWIwsE+QZmuiLzjzkUTBDNZ4QAstbN57MY4DrDDe3 -HbSKoCDrPWp+eWmuimD2U1fhxMpPKaUonXLvH1jAhsLYNUt+yS7o0UOlFm+gbktK -h2t4T+t2QeHzefkh8+Tud1EV0wObTbel54qda3LKGwgGUwPCMs7UxZ/oeSJTd25E -oKAizymEcdcnpPHBYfq7pr/QhPW2bpO1OVjVKBk8mN3SjVO1366MawxrAfr1AKby -wQcy8kVN2YskM0VSYl+Ck5BRJl//NO59oGFIPtp7IXy0F/yqokwkVJC4I7YPM3ua -r5Ozixl8+NFDPE+8SVrmUASiHJLbjG7RBT3SyWuzEMV642uQY6FHp6NMlncBczkK -slkXz/U5i+E/GM+5Q+MZ7HmwzrCmicYgqiUI8pryYLLPxRjtpiZ9f074jiRtXxFO -5HRZFackUg0Zz2e0pBhv8FCfLvMc4NiCt8LvqmcPNnXEjMc6m4/Vz+6UheLRBRK+
[GitHub] merlimat commented on issue #1156: Introduce ActiveConsumerListener for realizing if a consumer is active in a failover subscription group
merlimat commented on issue #1156: Introduce ActiveConsumerListener for realizing if a consumer is active in a failover subscription group URL: https://github.com/apache/incubator-pulsar/pull/1156#issuecomment-365801659 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on issue #1156: Introduce ActiveConsumerListener for realizing if a consumer is active in a failover subscription group
merlimat commented on issue #1156: Introduce ActiveConsumerListener for realizing if a consumer is active in a failover subscription group URL: https://github.com/apache/incubator-pulsar/pull/1156#issuecomment-365801659 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch branch-1.22 updated: add httpcore dep required by httpclient in hostname-verification (#1239)
This is an automated email from the ASF dual-hosted git repository. rdhabalia pushed a commit to branch branch-1.22 in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git The following commit(s) were added to refs/heads/branch-1.22 by this push: new 96d720c add httpcore dep required by httpclient in hostname-verification (#1239) 96d720c is described below commit 96d720c928b21485df8e055dde379aeeb7f81624 Author: Rajan DhabaliaAuthorDate: Wed Feb 14 16:29:37 2018 -0800 add httpcore dep required by httpclient in hostname-verification (#1239) --- all/src/assemble/LICENSE.bin.txt | 1 + pulsar-broker-shaded/pom.xml | 7 +++ pulsar-client-kafka-compat/pulsar-client-kafka/pom.xml | 9 + pulsar-client-shaded/pom.xml | 7 +++ pulsar-client/pom.xml | 7 ++- 5 files changed, 30 insertions(+), 1 deletion(-) diff --git a/all/src/assemble/LICENSE.bin.txt b/all/src/assemble/LICENSE.bin.txt index a7e70de..944ba2d 100644 --- a/all/src/assemble/LICENSE.bin.txt +++ b/all/src/assemble/LICENSE.bin.txt @@ -333,6 +333,7 @@ The Apache Software License, Version 2.0 * SnakeYaml -- org.yaml-snakeyaml-*.jar * RocksDB - org.rocksdb.*.jar * HttpClient - org.apache.httpcomponents.httpclient.jar + * HttCore - org.apache.httpcomponents.httpcore.jar * CommonsLogging - commons-logging-*.jar BSD 3-clause "New" or "Revised" License diff --git a/pulsar-broker-shaded/pom.xml b/pulsar-broker-shaded/pom.xml index 5794983..6d552bf 100644 --- a/pulsar-broker-shaded/pom.xml +++ b/pulsar-broker-shaded/pom.xml @@ -106,6 +106,7 @@ com.wordnik:swagger-annotations org.apache.httpcomponents:httpclient commons-logging:commons-logging + org.apache.httpcomponents:httpcore @@ -121,6 +122,12 @@ ** + + commons-logging:commons-logging + + ** + + diff --git a/pulsar-client-kafka-compat/pulsar-client-kafka/pom.xml b/pulsar-client-kafka-compat/pulsar-client-kafka/pom.xml index fe601bf..ea4b71c 100644 --- a/pulsar-client-kafka-compat/pulsar-client-kafka/pom.xml +++ b/pulsar-client-kafka-compat/pulsar-client-kafka/pom.xml @@ -127,8 +127,17 @@ com.yahoo.datasketches:sketches-core org.apache.httpcomponents:httpclient commons-logging:commons-logging + org.apache.httpcomponents:httpcore + + + commons-logging:commons-logging + + ** + + + org.apache.kafka.clients.producer.KafkaProducer diff --git a/pulsar-client-shaded/pom.xml b/pulsar-client-shaded/pom.xml index bebc36d..81f873e 100644 --- a/pulsar-client-shaded/pom.xml +++ b/pulsar-client-shaded/pom.xml @@ -83,6 +83,7 @@ com.yahoo.datasketches:sketches-core org.apache.httpcomponents:httpclient commons-logging:commons-logging + org.apache.httpcomponents:httpcore @@ -98,6 +99,12 @@ ** + + commons-logging:commons-logging + + ** + + diff --git a/pulsar-client/pom.xml b/pulsar-client/pom.xml index df6cc8d..abd0300 100644 --- a/pulsar-client/pom.xml +++ b/pulsar-client/pom.xml @@ -86,12 +86,17 @@ - + commons-logging commons-logging 1.1.1 + + org.apache.httpcomponents + httpcore + 4.4.9 + -- To stop receiving notification emails like this one, please contact rdhaba...@apache.org.
[GitHub] merlimat closed pull request #1239: add httpcore dep required by httpclient in hostname-verification
merlimat closed pull request #1239: add httpcore dep required by httpclient in hostname-verification URL: https://github.com/apache/incubator-pulsar/pull/1239 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/all/src/assemble/LICENSE.bin.txt b/all/src/assemble/LICENSE.bin.txt index a7e70defb..944ba2d1a 100644 --- a/all/src/assemble/LICENSE.bin.txt +++ b/all/src/assemble/LICENSE.bin.txt @@ -333,6 +333,7 @@ The Apache Software License, Version 2.0 * SnakeYaml -- org.yaml-snakeyaml-*.jar * RocksDB - org.rocksdb.*.jar * HttpClient - org.apache.httpcomponents.httpclient.jar + * HttCore - org.apache.httpcomponents.httpcore.jar * CommonsLogging - commons-logging-*.jar BSD 3-clause "New" or "Revised" License diff --git a/pulsar-broker-shaded/pom.xml b/pulsar-broker-shaded/pom.xml index efcfa76dc..e8539e751 100644 --- a/pulsar-broker-shaded/pom.xml +++ b/pulsar-broker-shaded/pom.xml @@ -106,6 +106,7 @@ com.wordnik:swagger-annotations org.apache.httpcomponents:httpclient commons-logging:commons-logging + org.apache.httpcomponents:httpcore @@ -121,6 +122,12 @@ ** + + commons-logging:commons-logging + + ** + + diff --git a/pulsar-client-kafka-compat/pulsar-client-kafka/pom.xml b/pulsar-client-kafka-compat/pulsar-client-kafka/pom.xml index 906000372..1be125024 100644 --- a/pulsar-client-kafka-compat/pulsar-client-kafka/pom.xml +++ b/pulsar-client-kafka-compat/pulsar-client-kafka/pom.xml @@ -127,8 +127,17 @@ com.yahoo.datasketches:sketches-core org.apache.httpcomponents:httpclient commons-logging:commons-logging + org.apache.httpcomponents:httpcore + + + commons-logging:commons-logging + + ** + + + org.apache.kafka.clients.producer.KafkaProducer diff --git a/pulsar-client-shaded/pom.xml b/pulsar-client-shaded/pom.xml index cf1d7b567..58bba0e52 100644 --- a/pulsar-client-shaded/pom.xml +++ b/pulsar-client-shaded/pom.xml @@ -83,6 +83,7 @@ com.yahoo.datasketches:sketches-core org.apache.httpcomponents:httpclient commons-logging:commons-logging + org.apache.httpcomponents:httpcore @@ -98,6 +99,12 @@ ** + + commons-logging:commons-logging + + ** + + diff --git a/pulsar-client/pom.xml b/pulsar-client/pom.xml index 57c69c8ea..3ab403fbe 100644 --- a/pulsar-client/pom.xml +++ b/pulsar-client/pom.xml @@ -86,12 +86,17 @@ - + commons-logging commons-logging 1.1.1 + + org.apache.httpcomponents + httpcore + 4.4.9 + This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch master updated: add httpcore dep required by httpclient in hostname-verification (#1239)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git The following commit(s) were added to refs/heads/master by this push: new 350d297 add httpcore dep required by httpclient in hostname-verification (#1239) 350d297 is described below commit 350d2973703f99f3a36e6a56a4348a6b76415c3f Author: Rajan DhabaliaAuthorDate: Wed Feb 14 16:29:37 2018 -0800 add httpcore dep required by httpclient in hostname-verification (#1239) --- all/src/assemble/LICENSE.bin.txt | 1 + pulsar-broker-shaded/pom.xml | 7 +++ pulsar-client-kafka-compat/pulsar-client-kafka/pom.xml | 9 + pulsar-client-shaded/pom.xml | 7 +++ pulsar-client/pom.xml | 7 ++- 5 files changed, 30 insertions(+), 1 deletion(-) diff --git a/all/src/assemble/LICENSE.bin.txt b/all/src/assemble/LICENSE.bin.txt index a7e70de..944ba2d 100644 --- a/all/src/assemble/LICENSE.bin.txt +++ b/all/src/assemble/LICENSE.bin.txt @@ -333,6 +333,7 @@ The Apache Software License, Version 2.0 * SnakeYaml -- org.yaml-snakeyaml-*.jar * RocksDB - org.rocksdb.*.jar * HttpClient - org.apache.httpcomponents.httpclient.jar + * HttCore - org.apache.httpcomponents.httpcore.jar * CommonsLogging - commons-logging-*.jar BSD 3-clause "New" or "Revised" License diff --git a/pulsar-broker-shaded/pom.xml b/pulsar-broker-shaded/pom.xml index efcfa76..e8539e7 100644 --- a/pulsar-broker-shaded/pom.xml +++ b/pulsar-broker-shaded/pom.xml @@ -106,6 +106,7 @@ com.wordnik:swagger-annotations org.apache.httpcomponents:httpclient commons-logging:commons-logging + org.apache.httpcomponents:httpcore @@ -121,6 +122,12 @@ ** + + commons-logging:commons-logging + + ** + + diff --git a/pulsar-client-kafka-compat/pulsar-client-kafka/pom.xml b/pulsar-client-kafka-compat/pulsar-client-kafka/pom.xml index 9060003..1be1250 100644 --- a/pulsar-client-kafka-compat/pulsar-client-kafka/pom.xml +++ b/pulsar-client-kafka-compat/pulsar-client-kafka/pom.xml @@ -127,8 +127,17 @@ com.yahoo.datasketches:sketches-core org.apache.httpcomponents:httpclient commons-logging:commons-logging + org.apache.httpcomponents:httpcore + + + commons-logging:commons-logging + + ** + + + org.apache.kafka.clients.producer.KafkaProducer diff --git a/pulsar-client-shaded/pom.xml b/pulsar-client-shaded/pom.xml index cf1d7b5..58bba0e 100644 --- a/pulsar-client-shaded/pom.xml +++ b/pulsar-client-shaded/pom.xml @@ -83,6 +83,7 @@ com.yahoo.datasketches:sketches-core org.apache.httpcomponents:httpclient commons-logging:commons-logging + org.apache.httpcomponents:httpcore @@ -98,6 +99,12 @@ ** + + commons-logging:commons-logging + + ** + + diff --git a/pulsar-client/pom.xml b/pulsar-client/pom.xml index 57c69c8..3ab403f 100644 --- a/pulsar-client/pom.xml +++ b/pulsar-client/pom.xml @@ -86,12 +86,17 @@ - + commons-logging commons-logging 1.1.1 + + org.apache.httpcomponents + httpcore + 4.4.9 + -- To stop receiving notification emails like this one, please contact mme...@apache.org.
[GitHub] sijie opened a new issue #1240: Implement ActiveConsumerListener for pulsar cpp client
sijie opened a new issue #1240: Implement ActiveConsumerListener for pulsar cpp client URL: https://github.com/apache/incubator-pulsar/issues/1240 ActiveConsumerListener is introduced in #1156 . We need similar feature implemented in cpp client. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #1156: Introduce ActiveConsumerListener for realizing if a consumer is active in a failover subscription group
sijie commented on issue #1156: Introduce ActiveConsumerListener for realizing if a consumer is active in a failover subscription group URL: https://github.com/apache/incubator-pulsar/pull/1156#issuecomment-365780104 the CI failed with license header. fixed license header. I forgot another item to make CPP client ignore this notification. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch master updated: Make max clients per topic/subscription configurable (#1234)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git The following commit(s) were added to refs/heads/master by this push: new 5613c2b Make max clients per topic/subscription configurable (#1234) 5613c2b is described below commit 5613c2b92f6b7989d6a4c65e259713953e933e8f Author: hrsakaiAuthorDate: Thu Feb 15 08:00:23 2018 +0900 Make max clients per topic/subscription configurable (#1234) --- conf/broker.conf | 15 ++ conf/standalone.conf | 15 ++ .../apache/pulsar/broker/ServiceConfiguration.java | 40 - .../AbstractDispatcherSingleActiveConsumer.java| 27 ++-- .../broker/service/BrokerServiceException.java | 8 +- .../NonPersistentDispatcherMultipleConsumers.java | 39 - ...onPersistentDispatcherSingleActiveConsumer.java | 22 ++- .../service/nonpersistent/NonPersistentTopic.java | 23 ++- .../PersistentDispatcherMultipleConsumers.java | 29 +++- .../PersistentDispatcherSingleActiveConsumer.java | 16 ++ .../broker/service/persistent/PersistentTopic.java | 22 +++ .../pulsar/broker/service/PersistentTopicTest.java | 172 - 12 files changed, 406 insertions(+), 22 deletions(-) diff --git a/conf/broker.conf b/conf/broker.conf index d18ebac..dbea41d 100644 --- a/conf/broker.conf +++ b/conf/broker.conf @@ -168,6 +168,21 @@ enableRunBookieTogether=false # Enable to run bookie autorecovery along with broker enableRunBookieAutoRecoveryTogether=false +// Max number of producers allowed to connect to topic. Once this limit reaches, Broker will reject new producers +// until the number of connected producers decrease. +// Using a value of 0, is disabling maxProducersPerTopic-limit check. +maxProducersPerTopic=0 + +// Max number of consumers allowed to connect to topic. Once this limit reaches, Broker will reject new consumers +// until the number of connected consumers decrease. +// Using a value of 0, is disabling maxConsumersPerTopic-limit check. +maxConsumersPerTopic=0 + +// Max number of consumers allowed to connect to subscription. Once this limit reaches, Broker will reject new consumers +// until the number of connected consumers decrease. +// Using a value of 0, is disabling maxConsumersPerSubscription-limit check. +maxConsumersPerSubscription=0 + ### --- Authentication --- ### # Role names that are treated as "proxy roles". If the broker sees a request with #role as proxyRoles - it will demand to see a valid original principal. diff --git a/conf/standalone.conf b/conf/standalone.conf index 1201dce..a6c915e 100644 --- a/conf/standalone.conf +++ b/conf/standalone.conf @@ -151,6 +151,21 @@ enablePersistentTopics=true # Enable broker to load non-persistent topics enableNonPersistentTopics=true +// Max number of producers allowed to connect to topic. Once this limit reaches, Broker will reject new producers +// until the number of connected producers decrease. +// Using a value of 0, is disabling maxProducersPerTopic-limit check. +maxProducersPerTopic=0 + +// Max number of consumers allowed to connect to topic. Once this limit reaches, Broker will reject new consumers +// until the number of connected consumers decrease. +// Using a value of 0, is disabling maxConsumersPerTopic-limit check. +maxConsumersPerTopic=0 + +// Max number of consumers allowed to connect to subscription. Once this limit reaches, Broker will reject new consumers +// until the number of connected consumers decrease. +// Using a value of 0, is disabling maxConsumersPerSubscription-limit check. +maxConsumersPerSubscription=0 + ### --- Authentication --- ### # Role names that are treated as "proxy roles". If the broker sees a request with #role as proxyRoles - it will demand to see a valid original principal. diff --git a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java index fcd9a92..4be2195 100644 --- a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java +++ b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java @@ -169,6 +169,21 @@ public class ServiceConfiguration implements PulsarConfiguration { // Enable to run bookie autorecovery along with broker private boolean enableRunBookieAutoRecoveryTogether = false; +// Max number of producers allowed to connect to topic. Once this limit reaches, Broker will reject new producers +// until the number of connected producers decrease. +// Using a value of 0, is disabling maxProducersPerTopic-limit check. +private int maxProducersPerTopic = 0; + +// Max number of consumers allowed to connect to topic. Once this limit reaches, Broker will reject new consumers +//
[GitHub] rdhabalia opened a new pull request #1239: add httpcore dep required by httpclient in hostname-verification
rdhabalia opened a new pull request #1239: add httpcore dep required by httpclient in hostname-verification URL: https://github.com/apache/incubator-pulsar/pull/1239 ### Motivation - when a broker's tls certificate present, httpclient's DefaultHostnameVerifier requires httpcore dependencies to use `org.apache.http.util.Args` class. ``` 11:12:33.486 [pulsar-discovery-io-2-3] WARN o.a.p.p.server.DirectProxyHandler- [[id: 0x300e8efa, L:/10.213.246.124:6651 - R:/10.215.134.167:51374]] [[id: 0x2b69265f, L:/10.213.246.124:48686 - R:perbs java.lang.NoClassDefFoundError: org/apache/http/util/Args at org.apache.http.conn.ssl.SubjectName.(SubjectName.java:48) ~[httpclient-4.5.5.jar:4.5.5] at org.apache.http.conn.ssl.DefaultHostnameVerifier.getSubjectAltNames(DefaultHostnameVerifier.java:310) ~[httpclient-4.5.5.jar:4.5.5] at org.apache.http.conn.ssl.DefaultHostnameVerifier.verify(DefaultHostnameVerifier.java:112) ~[httpclient-4.5.5.jar:4.5.5] at org.apache.http.conn.ssl.DefaultHostnameVerifier.verify(DefaultHostnameVerifier.java:99) ~[httpclient-4.5.5.jar:4.5.5] at ``` - Also `DefaultHostnameVerifier` requires all classes of commons-logging. ### Modifications add httpcore dependencies into client/proxy where hostname verification happens. ### Result It fixes `java.lang.NoClassDefFoundError: org/apache/http/util/Args` while hostname verification. Note: I will cherry-pick in 1.22 branch as well. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on a change in pull request #1103: PIP-13-1/3: Provide `TopicsConsumer` to consume from several topics under same namespace
merlimat commented on a change in pull request #1103: PIP-13-1/3: Provide `TopicsConsumer` to consume from several topics under same namespace URL: https://github.com/apache/incubator-pulsar/pull/1103#discussion_r168322621 ## File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/TopicsConsumerImpl.java ## @@ -0,0 +1,881 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pulsar.client.impl; + +import static com.google.common.base.Preconditions.checkArgument; +import static com.google.common.base.Preconditions.checkState; + +import com.google.common.collect.Lists; +import java.util.Collection; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Set; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ConcurrentLinkedQueue; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; +import java.util.concurrent.locks.ReadWriteLock; +import java.util.concurrent.locks.ReentrantReadWriteLock; +import java.util.stream.Collectors; +import java.util.stream.IntStream; + +import org.apache.pulsar.client.api.Consumer; +import org.apache.pulsar.client.api.ConsumerConfiguration; +import org.apache.pulsar.client.api.Message; +import org.apache.pulsar.client.api.MessageId; +import org.apache.pulsar.client.api.PulsarClientException; +import org.apache.pulsar.client.api.SubscriptionType; +import org.apache.pulsar.client.util.ConsumerName; +import org.apache.pulsar.client.util.FutureUtil; +import org.apache.pulsar.common.api.proto.PulsarApi.CommandAck.AckType; +import org.apache.pulsar.common.naming.DestinationName; +import org.apache.pulsar.common.naming.NamespaceName; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class TopicsConsumerImpl extends ConsumerBase { + +// All topics should be in same namespace +protected NamespaceName namespaceName; + +// Map, when get do ACK, consumer will by find by topic name +private final ConcurrentHashMap consumers; Review comment: I would suggest to just rename `PartitionedConsumerImpl`, change the constructor to accept a list of topics (rather than the number of partitions) and switch to use `TopicMessageIdImpl`. I don't think that `UnAckedMessageTracker` usage would need to be changed. That would avoid to add a lot of new code and having to merge it later. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] rdhabalia closed pull request #1238: add httpcore dep required by httpclient in hostname-verification
rdhabalia closed pull request #1238: add httpcore dep required by httpclient in hostname-verification URL: https://github.com/apache/incubator-pulsar/pull/1238 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/all/src/assemble/LICENSE.bin.txt b/all/src/assemble/LICENSE.bin.txt index a7e70defb..944ba2d1a 100644 --- a/all/src/assemble/LICENSE.bin.txt +++ b/all/src/assemble/LICENSE.bin.txt @@ -333,6 +333,7 @@ The Apache Software License, Version 2.0 * SnakeYaml -- org.yaml-snakeyaml-*.jar * RocksDB - org.rocksdb.*.jar * HttpClient - org.apache.httpcomponents.httpclient.jar + * HttCore - org.apache.httpcomponents.httpcore.jar * CommonsLogging - commons-logging-*.jar BSD 3-clause "New" or "Revised" License diff --git a/pulsar-broker-shaded/pom.xml b/pulsar-broker-shaded/pom.xml index efcfa76dc..3b2650960 100644 --- a/pulsar-broker-shaded/pom.xml +++ b/pulsar-broker-shaded/pom.xml @@ -106,6 +106,7 @@ com.wordnik:swagger-annotations org.apache.httpcomponents:httpclient commons-logging:commons-logging + org.apache.httpcomponents:httpcore diff --git a/pulsar-client-kafka-compat/pulsar-client-kafka/pom.xml b/pulsar-client-kafka-compat/pulsar-client-kafka/pom.xml index 906000372..b1e5cad05 100644 --- a/pulsar-client-kafka-compat/pulsar-client-kafka/pom.xml +++ b/pulsar-client-kafka-compat/pulsar-client-kafka/pom.xml @@ -127,6 +127,7 @@ com.yahoo.datasketches:sketches-core org.apache.httpcomponents:httpclient commons-logging:commons-logging + org.apache.httpcomponents:httpcore diff --git a/pulsar-client-shaded/pom.xml b/pulsar-client-shaded/pom.xml index cf1d7b567..dd58d5dca 100644 --- a/pulsar-client-shaded/pom.xml +++ b/pulsar-client-shaded/pom.xml @@ -83,6 +83,7 @@ com.yahoo.datasketches:sketches-core org.apache.httpcomponents:httpclient commons-logging:commons-logging + org.apache.httpcomponents:httpcore diff --git a/pulsar-client/pom.xml b/pulsar-client/pom.xml index 57c69c8ea..3ab403fbe 100644 --- a/pulsar-client/pom.xml +++ b/pulsar-client/pom.xml @@ -86,12 +86,17 @@ - + commons-logging commons-logging 1.1.1 + + org.apache.httpcomponents + httpcore + 4.4.9 + This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mgodave commented on a change in pull request #1232: Schema registry (1/4)
mgodave commented on a change in pull request #1232: Schema registry (1/4) URL: https://github.com/apache/incubator-pulsar/pull/1232#discussion_r168294644 ## File path: pulsar-common/src/main/proto/PulsarApi.proto ## @@ -368,6 +387,7 @@ message CommandRedeliverUnacknowledgedMessages { message CommandSuccess { required uint64 request_id = 1; + optional Schema schema = 2; Review comment: OK. I had originally proposed that but it was eliminated based on some input but I don't recall the specific circumstances. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch master updated: Add non-persistent topic stats separately in brokers-stat (#1235)
This is an automated email from the ASF dual-hosted git repository. rdhabalia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git The following commit(s) were added to refs/heads/master by this push: new 3c9e281 Add non-persistent topic stats separately in brokers-stat (#1235) 3c9e281 is described below commit 3c9e28172d486bfb4c39db5248e1033d5079c48f Author: Rajan DhabaliaAuthorDate: Wed Feb 14 12:03:38 2018 -0800 Add non-persistent topic stats separately in brokers-stat (#1235) --- .../apache/pulsar/broker/service/PulsarStats.java | 45 +- 1 file changed, 36 insertions(+), 9 deletions(-) diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PulsarStats.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PulsarStats.java index cdc5ada..835cabc 100644 --- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PulsarStats.java +++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PulsarStats.java @@ -25,6 +25,7 @@ import java.util.concurrent.locks.ReentrantReadWriteLock; import java.util.function.Consumer; import org.apache.pulsar.broker.PulsarService; +import org.apache.pulsar.broker.service.nonpersistent.NonPersistentTopic; import org.apache.pulsar.broker.service.persistent.PersistentTopic; import org.apache.pulsar.broker.stats.BrokerOperabilityMetrics; import org.apache.pulsar.broker.stats.ClusterReplicationMetrics; @@ -55,6 +56,7 @@ public class PulsarStats implements Closeable { private Map bundleStats; private List tempMetricsCollection; private List metricsCollection; +private List tempNonPersistentTopics; private final BrokerOperabilityMetrics brokerOperabilityMetrics; private final ReentrantReadWriteLock bufferLock = new ReentrantReadWriteLock(); @@ -71,6 +73,7 @@ public class PulsarStats implements Closeable { this.metricsCollection = Lists.newArrayList(); this.brokerOperabilityMetrics = new BrokerOperabilityMetrics(pulsar.getConfiguration().getClusterName(), pulsar.getAdvertisedAddress()); +this.tempNonPersistentTopics = Lists.newArrayList(); } @Override @@ -118,22 +121,46 @@ public class PulsarStats implements Closeable { currentBundleStats.topics = topics.size(); topicStatsStream.startObject(NamespaceBundle.getBundleRange(bundle)); + +tempNonPersistentTopics.clear(); +// start persistent topic topicStatsStream.startObject("persistent"); topics.forEach((name, topic) -> { -try { -topic.updateRates(nsStats, currentBundleStats, topicStatsStream, -clusterReplicationMetrics, namespaceName); -} catch (Exception e) { -log.error("Failed to generate topic stats for topic {}: {}", name, e.getMessage(), e); -} -// this task: helps to activate inactive-backlog-cursors which have caught up and -// connected, also deactivate active-backlog-cursors which has backlog if (topic instanceof PersistentTopic) { +try { +topic.updateRates(nsStats, currentBundleStats, topicStatsStream, +clusterReplicationMetrics, namespaceName); +} catch (Exception e) { +log.error("Failed to generate topic stats for topic {}: {}", name, e.getMessage(), e); +} +// this task: helps to activate inactive-backlog-cursors which have caught up and +// connected, also deactivate active-backlog-cursors which has backlog ((PersistentTopic) topic).getManagedLedger().checkBackloggedCursors(); +}else if (topic instanceof NonPersistentTopic) { + tempNonPersistentTopics.add((NonPersistentTopic) topic); +} else { +log.warn("Unsupported type of topic {}", topic.getClass().getName()); } }); - +// end persistent topics section topicStatsStream.endObject(); + +if(!tempNonPersistentTopics.isEmpty()) { + // start non-persistent topic +topicStatsStream.startObject("non-persistent"); +
[GitHub] rdhabalia closed pull request #1235: Add non-persistent topic stats separately in brokers-stat
rdhabalia closed pull request #1235: Add non-persistent topic stats separately in brokers-stat URL: https://github.com/apache/incubator-pulsar/pull/1235 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PulsarStats.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PulsarStats.java index cdc5ada99..835cabccd 100644 --- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PulsarStats.java +++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PulsarStats.java @@ -25,6 +25,7 @@ import java.util.function.Consumer; import org.apache.pulsar.broker.PulsarService; +import org.apache.pulsar.broker.service.nonpersistent.NonPersistentTopic; import org.apache.pulsar.broker.service.persistent.PersistentTopic; import org.apache.pulsar.broker.stats.BrokerOperabilityMetrics; import org.apache.pulsar.broker.stats.ClusterReplicationMetrics; @@ -55,6 +56,7 @@ private MapbundleStats; private List tempMetricsCollection; private List metricsCollection; +private List tempNonPersistentTopics; private final BrokerOperabilityMetrics brokerOperabilityMetrics; private final ReentrantReadWriteLock bufferLock = new ReentrantReadWriteLock(); @@ -71,6 +73,7 @@ public PulsarStats(PulsarService pulsar) { this.metricsCollection = Lists.newArrayList(); this.brokerOperabilityMetrics = new BrokerOperabilityMetrics(pulsar.getConfiguration().getClusterName(), pulsar.getAdvertisedAddress()); +this.tempNonPersistentTopics = Lists.newArrayList(); } @Override @@ -118,22 +121,46 @@ public synchronized void updateStats( currentBundleStats.topics = topics.size(); topicStatsStream.startObject(NamespaceBundle.getBundleRange(bundle)); + +tempNonPersistentTopics.clear(); +// start persistent topic topicStatsStream.startObject("persistent"); topics.forEach((name, topic) -> { -try { -topic.updateRates(nsStats, currentBundleStats, topicStatsStream, -clusterReplicationMetrics, namespaceName); -} catch (Exception e) { -log.error("Failed to generate topic stats for topic {}: {}", name, e.getMessage(), e); -} -// this task: helps to activate inactive-backlog-cursors which have caught up and -// connected, also deactivate active-backlog-cursors which has backlog if (topic instanceof PersistentTopic) { +try { +topic.updateRates(nsStats, currentBundleStats, topicStatsStream, +clusterReplicationMetrics, namespaceName); +} catch (Exception e) { +log.error("Failed to generate topic stats for topic {}: {}", name, e.getMessage(), e); +} +// this task: helps to activate inactive-backlog-cursors which have caught up and +// connected, also deactivate active-backlog-cursors which has backlog ((PersistentTopic) topic).getManagedLedger().checkBackloggedCursors(); +}else if (topic instanceof NonPersistentTopic) { + tempNonPersistentTopics.add((NonPersistentTopic) topic); +} else { +log.warn("Unsupported type of topic {}", topic.getClass().getName()); } }); - +// end persistent topics section topicStatsStream.endObject(); + +if(!tempNonPersistentTopics.isEmpty()) { + // start non-persistent topic +topicStatsStream.startObject("non-persistent"); +tempNonPersistentTopics.forEach(topic -> { +try { +topic.updateRates(nsStats, currentBundleStats, topicStatsStream, +clusterReplicationMetrics, namespaceName); +} catch (Exception e) { +log.error("Failed to generate
[GitHub] rdhabalia opened a new pull request #1238: add httpcore dep required by httpclient in hostname-verification
rdhabalia opened a new pull request #1238: add httpcore dep required by httpclient in hostname-verification URL: https://github.com/apache/incubator-pulsar/pull/1238 ### Motivation when a broker's tls certificate present, httpclient's DefaultHostnameVerifier requires httpcore dependencies to use `org.apache.http.util.Args` class. ``` 11:12:33.486 [pulsar-discovery-io-2-3] WARN o.a.p.p.server.DirectProxyHandler- [[id: 0x300e8efa, L:/10.213.246.124:6651 - R:/10.215.134.167:51374]] [[id: 0x2b69265f, L:/10.213.246.124:48686 - R:perbs java.lang.NoClassDefFoundError: org/apache/http/util/Args at org.apache.http.conn.ssl.SubjectName.(SubjectName.java:48) ~[httpclient-4.5.5.jar:4.5.5] at org.apache.http.conn.ssl.DefaultHostnameVerifier.getSubjectAltNames(DefaultHostnameVerifier.java:310) ~[httpclient-4.5.5.jar:4.5.5] at org.apache.http.conn.ssl.DefaultHostnameVerifier.verify(DefaultHostnameVerifier.java:112) ~[httpclient-4.5.5.jar:4.5.5] at org.apache.http.conn.ssl.DefaultHostnameVerifier.verify(DefaultHostnameVerifier.java:99) ~[httpclient-4.5.5.jar:4.5.5] at ``` ### Modifications add httpcore dependencies into client/proxy where hostname verification happens. ### Result It fixes `java.lang.NoClassDefFoundError: org/apache/http/util/Args` while hostname verification. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on a change in pull request #1232: Schema registry (1/4)
merlimat commented on a change in pull request #1232: Schema registry (1/4) URL: https://github.com/apache/incubator-pulsar/pull/1232#discussion_r168293225 ## File path: pulsar-common/src/main/proto/PulsarApi.proto ## @@ -368,6 +387,7 @@ message CommandRedeliverUnacknowledgedMessages { message CommandSuccess { required uint64 request_id = 1; + optional Schema schema = 2; Review comment: I think it would make sense to have a schema at the namespace level (with per-topic override), since we already have a lot of administration tasks at that level. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mgodave commented on a change in pull request #1232: Schema registry (1/4)
mgodave commented on a change in pull request #1232: Schema registry (1/4) URL: https://github.com/apache/incubator-pulsar/pull/1232#discussion_r168283925 ## File path: pulsar-common/src/main/proto/PulsarApi.proto ## @@ -368,6 +387,7 @@ message CommandRedeliverUnacknowledgedMessages { message CommandSuccess { required uint64 request_id = 1; + optional Schema schema = 2; Review comment: @dave2wave I've been thinking about this more. If this is a common use case I think I can make this happen with little upheaval. Let me know if this is a deal breaker for you. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mgodave commented on a change in pull request #1232: Schema registry (1/4)
mgodave commented on a change in pull request #1232: Schema registry (1/4) URL: https://github.com/apache/incubator-pulsar/pull/1232#discussion_r168277614 ## File path: pulsar-common/src/main/proto/PulsarApi.proto ## @@ -368,6 +387,7 @@ message CommandRedeliverUnacknowledgedMessages { message CommandSuccess { required uint64 request_id = 1; + optional Schema schema = 2; Review comment: That is not impossible, but perhaps inconvenient. What is the use case here? Are we talking within the same namespace or across namespaces? It would be possible to add another layer of indirection to the schema and store the name associated with the topic somewhere (not sure where at the moment). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mgodave commented on a change in pull request #1232: Schema registry (1/4)
mgodave commented on a change in pull request #1232: Schema registry (1/4) URL: https://github.com/apache/incubator-pulsar/pull/1232#discussion_r168275516 ## File path: pulsar-common/src/main/proto/PulsarApi.proto ## @@ -79,8 +86,9 @@ message MessageMetadata { repeated EncryptionKeys encryption_keys = 13; // Algorithm used to encrypt data key optional string encryption_algo = 14; + optional bytes schema_version = 15; // Additional parameters required by encryption - optional bytes encryption_param = 15; + optional bytes encryption_param = 16; Review comment: Sorry, I missed that. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mgodave commented on a change in pull request #1232: Schema registry (1/4)
mgodave commented on a change in pull request #1232: Schema registry (1/4) URL: https://github.com/apache/incubator-pulsar/pull/1232#discussion_r168275383 ## File path: pulsar-common/src/main/proto/PulsarApi.proto ## @@ -22,6 +22,20 @@ package pulsar.proto; option java_package = "org.apache.pulsar.common.api.proto"; option optimize_for = LITE_RUNTIME; +message Schema { + required string name = 1; + required bytes version = 2; + required bytes schema_data = 3; Review comment: The properties list is available to store that info. This makes the most sense to me but I'm willing to change it if you are dead set on specifying this information statically. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on a change in pull request #1232: Schema registry (1/4)
merlimat commented on a change in pull request #1232: Schema registry (1/4) URL: https://github.com/apache/incubator-pulsar/pull/1232#discussion_r168274945 ## File path: pulsar-common/src/main/proto/PulsarApi.proto ## @@ -79,8 +86,9 @@ message MessageMetadata { repeated EncryptionKeys encryption_keys = 13; // Algorithm used to encrypt data key optional string encryption_algo = 14; + optional bytes schema_version = 15; // Additional parameters required by encryption - optional bytes encryption_param = 15; + optional bytes encryption_param = 16; Review comment: We cannot change the integer id of an existing field This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch branch-1.20-yahoo deleted (was d5175b9)
This is an automated email from the ASF dual-hosted git repository. jai1 pushed a change to branch branch-1.20-yahoo in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git. was d5175b9 Bumped to 1.20.8-incubating-yahoo-SNAPSHOT This change permanently discards the following revisions: discard d5175b9 Bumped to 1.20.8-incubating-yahoo-SNAPSHOT discard 870268e Cancel keep-alive timer task after the proxy switch to TCP proxy (#1210) discard 7c460ac Reset kafka dep (#17) discard 02ec84b Bumped to 1.20.7-incubating-yahoo discard 8ae92d3 CMS-2984: Enable specification of TLS Protocol Versions and Cipher Suites discard d8f6068 Use SecurityUtility class (#16) discard 136c241 Support hostname verification on proxy to broker connection (#1214) (#14) discard 1336b85 Add hostname-verification at client tls connection (#1208) (#13) discard a86e472 Pulsar proxy (#11) discard 5c35550 Bumped to 1.20.7-incubating-yahoo-SNAPSHOT discard fdd255a Release to 1.20.6-incubating-yahoo discard 1b2d720 Add timeout for bundle unloading (#990) discard c24f03a Bumped to 1.20.6-incubating-yahoo-SNAPSHOT discard ab4b30d Release to 1.20.5-incubating-yahoo discard 031d64a add NoSuchEntryException as not-recoverable error (#10) discard fc954a0 Bumped up 1.20.5-incubating-yahoo-SNAPSHOT discard 3cbdb76 Release pulsar to 1.20.4-incubating-yahoo discard 3773830 Make sure nextTuple emits tuple with non-null values #1000 discard 61c6d75 Fix: cursor-ledger leak by deleting cursor-ledger after recovering managed-cursor discard 8a1132a Bumped to 1.20.4-incubating-yahoo-SNAPSHOT discard 403a409 Release pulsar to 1.20.3-incubating-yahoo discard 9b0a98d shade data-sketch (#8) discard 58f Bumped to 1.20.3-incubating-yahoo-SNAPSHOT discard 0b15038 Release pulsar to 1.20.2-incubating-yahoo discard 5eba7c1 Avoid dead-lock on broker service when broker failed to load managed-ledger (#934) discard 90be3c7 Fix: failed producer creation leak (#927) discard 05e6a20 Bumped to 1.20.2-incubating-yahoo-SNAPSHOT discard 67e0b88 Release pulsar to 1.20.1-incubating-yahoo discard dda80f8 Log incompatible client version on partition-metadata access discard 65722cc Fail broker start if broker-znode created by other zk-session (#913) discard 05742e6 PIP-8: Introduce peer cluster for global namespace redirection (#903) discard 564d1cc Avoid logging stacktrace for exclusive consumer-busy exception (#906) discard b979691 Cleanup unsused ApiVersionFilter (#885) discard c9c0a4b Specify rsa transform, aes & tag length (#839) discard 054cc49 Use thread local to allocate temp byte[] instead of allocator (#882) discard 1482cd0 Fix: Non-persistent topic acking sequence for dropped message (#874) discard 624e2f4 Bumped to 1.20.1-incubating-yahoo-SNAPSHOT discard eaf5587 Release pulsar to 1.20.0-incubating-yahoo -- To stop receiving notification emails like this one, please contact j...@apache.org.
[GitHub] merlimat commented on a change in pull request #1232: Schema registry (1/4)
merlimat commented on a change in pull request #1232: Schema registry (1/4) URL: https://github.com/apache/incubator-pulsar/pull/1232#discussion_r168274469 ## File path: pulsar-common/src/main/proto/PulsarApi.proto ## @@ -22,6 +22,20 @@ package pulsar.proto; option java_package = "org.apache.pulsar.common.api.proto"; option optimize_for = LITE_RUNTIME; +message Schema { + required string name = 1; + required bytes version = 2; + required bytes schema_data = 3; Review comment: I agree that storing the schema data is agnostic to the format, however these protobufs are relative to the interactions between client and broker. In this case, the producer might use something like `Schema.avroOf(MyClass.class)` or `Schema.jsonOf(MyClass.class)`. In this case, don't we need to communicate to the server what kind of schema the client is using? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dave2wave commented on a change in pull request #1232: Schema registry (1/4)
dave2wave commented on a change in pull request #1232: Schema registry (1/4) URL: https://github.com/apache/incubator-pulsar/pull/1232#discussion_r168274341 ## File path: pulsar-common/src/main/proto/PulsarApi.proto ## @@ -368,6 +387,7 @@ message CommandRedeliverUnacknowledgedMessages { message CommandSuccess { required uint64 request_id = 1; + optional Schema schema = 2; Review comment: I would want to assign a schema to multiple topics - perhaps a large number. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mgodave commented on a change in pull request #1232: Schema registry (1/4)
mgodave commented on a change in pull request #1232: Schema registry (1/4) URL: https://github.com/apache/incubator-pulsar/pull/1232#discussion_r168271879 ## File path: pulsar-common/src/main/proto/PulsarApi.proto ## @@ -22,6 +22,20 @@ package pulsar.proto; option java_package = "org.apache.pulsar.common.api.proto"; option optimize_for = LITE_RUNTIME; +message Schema { + required string name = 1; + required bytes version = 2; + required bytes schema_data = 3; Review comment: I tried to address this in a question posed on the original reference PR. I had originally specced this out with that intent but after much thought I decided not to add it. My intent was that the schema registry should just store a "Schema" no matter what it is, it does not need to know what it contains. The schema only really means something to the end systems (producer/consumer) and by extension the broker. We can allow the broker to "plug-in" modules to allow semantic parsing of the schema as needed. This way we can support any number of crazy business cases. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mgodave commented on a change in pull request #1232: Schema registry (1/4)
mgodave commented on a change in pull request #1232: Schema registry (1/4) URL: https://github.com/apache/incubator-pulsar/pull/1232#discussion_r168271025 ## File path: pulsar-common/src/main/proto/PulsarApi.proto ## @@ -22,6 +22,20 @@ package pulsar.proto; option java_package = "org.apache.pulsar.common.api.proto"; option optimize_for = LITE_RUNTIME; +message Schema { + required string name = 1; Review comment: grrr, I HATE TABS ;-) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ivankelly commented on a change in pull request #1232: Schema registry (1/4)
ivankelly commented on a change in pull request #1232: Schema registry (1/4) URL: https://github.com/apache/incubator-pulsar/pull/1232#discussion_r168262139 ## File path: pulsar-common/src/main/proto/PulsarApi.proto ## @@ -368,6 +387,7 @@ message CommandRedeliverUnacknowledgedMessages { message CommandSuccess { required uint64 request_id = 1; + optional Schema schema = 2; Review comment: ok, in that case, this makes sense. perhaps it should be a repeated rather than an optional though, for the case that there's different versions. I guess that the schema name must be immutable for a topic then also. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #1156: Introduce ActiveConsumerListener for realizing if a consumer is active in a failover subscription group
sijie commented on issue #1156: Introduce ActiveConsumerListener for realizing if a consumer is active in a failover subscription group URL: https://github.com/apache/incubator-pulsar/pull/1156#issuecomment-365688094 rebased to the latest master This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ivankelly commented on a change in pull request #1232: Schema registry (1/4)
ivankelly commented on a change in pull request #1232: Schema registry (1/4) URL: https://github.com/apache/incubator-pulsar/pull/1232#discussion_r168240669 ## File path: pulsar-common/src/main/proto/PulsarApi.proto ## @@ -368,6 +387,7 @@ message CommandRedeliverUnacknowledgedMessages { message CommandSuccess { required uint64 request_id = 1; + optional Schema schema = 2; Review comment: Will the whole schema go with all messages? If that's the case, isn't this field here redundant? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on a change in pull request #1089: PIP-12 Introduce builder for creating Producer Consumer Reader
merlimat commented on a change in pull request #1089: PIP-12 Introduce builder for creating Producer Consumer Reader URL: https://github.com/apache/incubator-pulsar/pull/1089#discussion_r168237399 ## File path: pulsar-client/src/main/java/org/apache/pulsar/client/api/ClientBuilder.java ## @@ -0,0 +1,263 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pulsar.client.api; + +import java.io.Serializable; +import java.util.Map; +import java.util.concurrent.TimeUnit; + +import org.apache.pulsar.client.api.PulsarClientException.UnsupportedAuthenticationException; + +/** + * Builder interface that is used to construct a {@link PulsarClient} instance. + */ +public interface ClientBuilder extends Serializable, Cloneable { + +/** + * @return the new {@link PulsarClient} instance + */ +PulsarClient build() throws PulsarClientException; + +/** + * Create a copy of the current client builder. + * + * Cloning the builder can be used to share an incomplete configuration and specialize it multiple times. For + * example: + * + * + * ClientBuilder builder = PulsarClient.builder().ioThreads(8).listenerThreads(4); + * + * PulsarClient client1 = builder.clone().serviceUrl(URL_1).build(); + * PulsarClient client2 = builder.clone().serviceUrl(URL_2).build(); + * + */ +ClientBuilder clone(); Review comment: `clone()` is a bit weird in the way it's treated. It cannot be marked as `@Override` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on a change in pull request #1089: PIP-12 Introduce builder for creating Producer Consumer Reader
merlimat commented on a change in pull request #1089: PIP-12 Introduce builder for creating Producer Consumer Reader URL: https://github.com/apache/incubator-pulsar/pull/1089#discussion_r168236993 ## File path: pulsar-client/src/main/java/org/apache/pulsar/client/api/ClientBuilder.java ## @@ -0,0 +1,263 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pulsar.client.api; + +import java.io.Serializable; +import java.util.Map; +import java.util.concurrent.TimeUnit; + +import org.apache.pulsar.client.api.PulsarClientException.UnsupportedAuthenticationException; + +/** + * Builder interface that is used to construct a {@link PulsarClient} instance. Review comment: Sure, will add `@since` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on a change in pull request #1089: PIP-12 Introduce builder for creating Producer Consumer Reader
merlimat commented on a change in pull request #1089: PIP-12 Introduce builder for creating Producer Consumer Reader URL: https://github.com/apache/incubator-pulsar/pull/1089#discussion_r168236725 ## File path: pulsar-client/src/main/java/org/apache/pulsar/client/api/ConsumerBuilder.java ## @@ -0,0 +1,230 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pulsar.client.api; + +import java.io.Serializable; +import java.util.Map; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.TimeUnit; + +/** + * {@link ConsumerBuilder} is used to configure and create instances of {@link Consumer}. + * + * @see PulsarClient#newConsumer() + */ +public interface ConsumerBuilder extends Serializable, Cloneable { + +/** + * Create a copy of the current consumer builder. + * + * Cloning the builder can be used to share an incomplete configuration and specialize it multiple times. For + * example: + * + * + * ConsumerBuilder builder = client.newConsumer() // + * .subscriptionName("my-subscription-name") // + * .subscriptionType(SubscriptionType.Shared) // + * .receiverQueueSize(10); + * + * Consumer consumer1 = builder.clone().topic(TOPIC_1).subscribe(); + * Consumer consumer2 = builder.clone().topic(TOPIC_2).subscribe(); + * + */ +ConsumerBuilder clone(); + +/** + * Finalize the consumer creation by subscribing to the topic. + * + * + * Subscribing to the c Review comment: ups, forgot to finish This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on a change in pull request #1232: Schema registry (1/4)
merlimat commented on a change in pull request #1232: Schema registry (1/4) URL: https://github.com/apache/incubator-pulsar/pull/1232#discussion_r168236331 ## File path: pulsar-common/src/main/proto/PulsarApi.proto ## @@ -272,6 +289,8 @@ message CommandProducer { /// Add optional metadata key=value to this producer repeated KeyValue metadata= 6; + + optional int64 schema_version = 7; Review comment: It should carry the whole schema here. The application/producer doesn't necessarely know what the version is. Eg: if I'm using a type like : ```java class MyClass { public int a; public int b; }; ``` When I try to publish, I just know the format I'm about to send, and we can extract the avro/json schema out of it. When I try to create producer, I can send that information, and the broker can compare with the schemas already availalable for this topic. If the schema is know, we just return the schema version in the `ProducerSuccess` response. If it's a new (compatible) version of the schema, it will also be stored. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #1156: Introduce ActiveConsumerListener for realizing if a consumer is active in a failover subscription group
sijie commented on issue #1156: Introduce ActiveConsumerListener for realizing if a consumer is active in a failover subscription group URL: https://github.com/apache/incubator-pulsar/pull/1156#issuecomment-365666967 > This could also have been done by each consumer periodically adding a message to the topic. if you start getting the messages, you are the active consumer. @ivankelly : yes that's the trick we are using now for the other project. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on a change in pull request #1089: PIP-12 Introduce builder for creating Producer Consumer Reader
sijie commented on a change in pull request #1089: PIP-12 Introduce builder for creating Producer Consumer Reader URL: https://github.com/apache/incubator-pulsar/pull/1089#discussion_r168229704 ## File path: pulsar-client/src/main/java/org/apache/pulsar/client/api/ClientBuilder.java ## @@ -0,0 +1,263 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pulsar.client.api; + +import java.io.Serializable; +import java.util.Map; +import java.util.concurrent.TimeUnit; + +import org.apache.pulsar.client.api.PulsarClientException.UnsupportedAuthenticationException; + +/** + * Builder interface that is used to construct a {@link PulsarClient} instance. Review comment: nit: it might be good to add "@since 2.0". so we can track when we introduce the build methods. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on a change in pull request #1089: PIP-12 Introduce builder for creating Producer Consumer Reader
sijie commented on a change in pull request #1089: PIP-12 Introduce builder for creating Producer Consumer Reader URL: https://github.com/apache/incubator-pulsar/pull/1089#discussion_r168230414 ## File path: pulsar-client/src/main/java/org/apache/pulsar/client/api/ConsumerBuilder.java ## @@ -0,0 +1,230 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pulsar.client.api; + +import java.io.Serializable; +import java.util.Map; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.TimeUnit; + +/** + * {@link ConsumerBuilder} is used to configure and create instances of {@link Consumer}. + * + * @see PulsarClient#newConsumer() + */ +public interface ConsumerBuilder extends Serializable, Cloneable { + +/** + * Create a copy of the current consumer builder. + * + * Cloning the builder can be used to share an incomplete configuration and specialize it multiple times. For + * example: + * + * + * ConsumerBuilder builder = client.newConsumer() // + * .subscriptionName("my-subscription-name") // + * .subscriptionType(SubscriptionType.Shared) // + * .receiverQueueSize(10); + * + * Consumer consumer1 = builder.clone().topic(TOPIC_1).subscribe(); + * Consumer consumer2 = builder.clone().topic(TOPIC_2).subscribe(); + * + */ +ConsumerBuilder clone(); + +/** + * Finalize the consumer creation by subscribing to the topic. + * + * + * Subscribing to the c Review comment: incomplete comment? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on a change in pull request #1089: PIP-12 Introduce builder for creating Producer Consumer Reader
sijie commented on a change in pull request #1089: PIP-12 Introduce builder for creating Producer Consumer Reader URL: https://github.com/apache/incubator-pulsar/pull/1089#discussion_r168230587 ## File path: pulsar-client/src/main/java/org/apache/pulsar/client/api/ConsumerBuilder.java ## @@ -0,0 +1,230 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pulsar.client.api; + +import java.io.Serializable; +import java.util.Map; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.TimeUnit; + +/** + * {@link ConsumerBuilder} is used to configure and create instances of {@link Consumer}. + * + * @see PulsarClient#newConsumer() + */ +public interface ConsumerBuilder extends Serializable, Cloneable { + +/** + * Create a copy of the current consumer builder. + * + * Cloning the builder can be used to share an incomplete configuration and specialize it multiple times. For + * example: + * + * + * ConsumerBuilder builder = client.newConsumer() // + * .subscriptionName("my-subscription-name") // + * .subscriptionType(SubscriptionType.Shared) // + * .receiverQueueSize(10); + * + * Consumer consumer1 = builder.clone().topic(TOPIC_1).subscribe(); + * Consumer consumer2 = builder.clone().topic(TOPIC_2).subscribe(); + * + */ +ConsumerBuilder clone(); + +/** + * Finalize the consumer creation by subscribing to the topic. + * + * + * Subscribing to the c + * + * @return Review comment: `@return a consumer`? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on a change in pull request #1089: PIP-12 Introduce builder for creating Producer Consumer Reader
sijie commented on a change in pull request #1089: PIP-12 Introduce builder for creating Producer Consumer Reader URL: https://github.com/apache/incubator-pulsar/pull/1089#discussion_r168231825 ## File path: pulsar-client/src/main/java/org/apache/pulsar/client/api/ReaderConfiguration.java ## @@ -84,8 +89,9 @@ public ReaderConfiguration setCryptoKeyReader(CryptoKeyReader cryptoKeyReader) { /** * Sets the ConsumerCryptoFailureAction to the value specified - * - * @param The consumer action + * + * @param The Review comment: the param field seems to be wrong? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on a change in pull request #1089: PIP-12 Introduce builder for creating Producer Consumer Reader
sijie commented on a change in pull request #1089: PIP-12 Introduce builder for creating Producer Consumer Reader URL: https://github.com/apache/incubator-pulsar/pull/1089#discussion_r168229917 ## File path: pulsar-client/src/main/java/org/apache/pulsar/client/api/ClientBuilder.java ## @@ -0,0 +1,263 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pulsar.client.api; + +import java.io.Serializable; +import java.util.Map; +import java.util.concurrent.TimeUnit; + +import org.apache.pulsar.client.api.PulsarClientException.UnsupportedAuthenticationException; + +/** + * Builder interface that is used to construct a {@link PulsarClient} instance. + */ +public interface ClientBuilder extends Serializable, Cloneable { + +/** + * @return the new {@link PulsarClient} instance + */ +PulsarClient build() throws PulsarClientException; + +/** + * Create a copy of the current client builder. + * + * Cloning the builder can be used to share an incomplete configuration and specialize it multiple times. For + * example: + * + * + * ClientBuilder builder = PulsarClient.builder().ioThreads(8).listenerThreads(4); + * + * PulsarClient client1 = builder.clone().serviceUrl(URL_1).build(); + * PulsarClient client2 = builder.clone().serviceUrl(URL_2).build(); + * + */ +ClientBuilder clone(); Review comment: nit: `@Override`? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ivankelly commented on a change in pull request #1232: Schema registry (1/4)
ivankelly commented on a change in pull request #1232: Schema registry (1/4) URL: https://github.com/apache/incubator-pulsar/pull/1232#discussion_r168196011 ## File path: pulsar-common/src/main/proto/PulsarApi.proto ## @@ -22,6 +22,20 @@ package pulsar.proto; option java_package = "org.apache.pulsar.common.api.proto"; option optimize_for = LITE_RUNTIME; +message Schema { + required string name = 1; Review comment: tabs This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ivankelly commented on a change in pull request #1232: Schema registry (1/4)
ivankelly commented on a change in pull request #1232: Schema registry (1/4) URL: https://github.com/apache/incubator-pulsar/pull/1232#discussion_r168230410 ## File path: pulsar-common/src/main/proto/PulsarApi.proto ## @@ -22,6 +22,20 @@ package pulsar.proto; option java_package = "org.apache.pulsar.common.api.proto"; option optimize_for = LITE_RUNTIME; +message Schema { + required string name = 1; + required bytes version = 2; + required bytes schema_data = 3; Review comment: Does the schema data itself have a schema? If we're supporting json, arvo, pb, etc, shouldn't we have a separate field to give the reader a hint of how they need to parse the schema? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ivankelly commented on a change in pull request #1232: Schema registry (1/4)
ivankelly commented on a change in pull request #1232: Schema registry (1/4) URL: https://github.com/apache/incubator-pulsar/pull/1232#discussion_r168229944 ## File path: pulsar-common/src/main/proto/PulsarApi.proto ## @@ -22,6 +22,20 @@ package pulsar.proto; option java_package = "org.apache.pulsar.common.api.proto"; option optimize_for = LITE_RUNTIME; +message Schema { + required string name = 1; + required bytes version = 2; + required bytes schema_data = 3; +repeated KeyValue properties = 4; +} + +message Tombstone {} + +message SchemaEntry { Review comment: I assume this is part of a the schema registry part? Why is tombstone separate from the schema? You have schema_data as required? If tombstoning a schema (which I assume is deleting the schema) what should the data be? Couldn't schema data itself being empty denote the tombstone? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ivankelly commented on a change in pull request #1232: Schema registry (1/4)
ivankelly commented on a change in pull request #1232: Schema registry (1/4) URL: https://github.com/apache/incubator-pulsar/pull/1232#discussion_r168231228 ## File path: pulsar-common/src/main/proto/PulsarApi.proto ## @@ -368,6 +387,7 @@ message CommandRedeliverUnacknowledgedMessages { message CommandSuccess { required uint64 request_id = 1; + optional Schema schema = 2; Review comment: what happens if the message schema changes mid topic? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on a change in pull request #1232: Schema registry (1/4)
merlimat commented on a change in pull request #1232: Schema registry (1/4) URL: https://github.com/apache/incubator-pulsar/pull/1232#discussion_r168231034 ## File path: pulsar-common/src/main/proto/PulsarApi.proto ## @@ -272,6 +289,8 @@ message CommandProducer { /// Add optional metadata key=value to this producer repeated KeyValue metadata= 6; + + optional int64 schema_version = 7; Review comment: Sure, no problem. I think it would be good to have the "final" version of the `PulsarApi.proto` to get a sense of all the changes in the interaction between client & broker. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on issue #1089: PIP-12 Introduce builder for creating Producer Consumer Reader
merlimat commented on issue #1089: PIP-12 Introduce builder for creating Producer Consumer Reader URL: https://github.com/apache/incubator-pulsar/pull/1089#issuecomment-365505550 > Once we have subscription for topics and pattern, we need add them in. Yes, the idea was to have different ways to specify: ```java // Single topic client.newConsumer().topic(MY_TOPIC).subscriptioName(SUB).build(); // List of topics List myList = ...; client.newConsumer().topics(myList).subscriptioName(SUB).build(); // Regex client.newConsumer().topicsRegex("test.*").subscriptioName(SUB).build(); ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on issue #1156: Introduce ActiveConsumerListener for realizing if a consumer is active in a failover subscription group
merlimat commented on issue #1156: Introduce ActiveConsumerListener for realizing if a consumer is active in a failover subscription group URL: https://github.com/apache/incubator-pulsar/pull/1156#issuecomment-365660980 > Does this provide a way to figure out the current active consumer? Not right now, I think it would be interesting to mention who's the active consumer in the topic in the topics stats. We could have a `Boolean` field in the `ConsumerStats` that is set to `null` in non-failover subscriptions. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #1223: Add a `backend` admin restful endpoint for query backend information
sijie commented on issue #1223: Add a `backend` admin restful endpoint for query backend information URL: https://github.com/apache/incubator-pulsar/pull/1223#issuecomment-365658827 @rdhabalia @yush1ga I renamed "backend" to "internalConfiguration". Can you guys review this change again? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ivankelly commented on a change in pull request #1156: Introduce ActiveConsumerListener for realizing if a consumer is active in a failover subscription group
ivankelly commented on a change in pull request #1156: Introduce ActiveConsumerListener for realizing if a consumer is active in a failover subscription group URL: https://github.com/apache/incubator-pulsar/pull/1156#discussion_r168221779 ## File path: pulsar-client/src/main/java/org/apache/pulsar/client/api/ConsumerEventListener.java ## @@ -0,0 +1,36 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pulsar.client.api; + +/** + * Listener on the consumer state changes. + */ +public interface ConsumerEventListener { + +/** + * Notified when the consumer group is changed, and the consumer becomes the active consumer. + */ +void becomeActive(Consumer consumer, int partitionId); Review comment: became? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat closed pull request #1228: Force to pull docker build image to regenerate PulsarApi.java generat?
merlimat closed pull request #1228: Force to pull docker build image to regenerate PulsarApi.java generat? URL: https://github.com/apache/incubator-pulsar/pull/1228 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/pulsar-common/generate_protobuf_docker.sh b/pulsar-common/generate_protobuf_docker.sh index c568c9a07..376d72e4c 100755 --- a/pulsar-common/generate_protobuf_docker.sh +++ b/pulsar-common/generate_protobuf_docker.sh @@ -32,6 +32,9 @@ IMAGE="$BUILD_IMAGE_NAME:$BUILD_IMAGE_VERSION" echo $IMAGE +# Force to pull image in case it was updated +docker pull $IMAGE + docker run -i \ -v ${COMMON_DIR}:${COMMON_DIR} $IMAGE \ bash -c "cd ${COMMON_DIR}; /pulsar/protobuf/src/protoc --java_out=src/main/java src/main/proto/PulsarApi.proto" This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #1186: Pulsar FAQ
sijie commented on issue #1186: Pulsar FAQ URL: https://github.com/apache/incubator-pulsar/pull/1186#issuecomment-365653665 @lucperkins do you have time on helping with the FAQ content here? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #1156: Introduce ActiveConsumerListener for realizing if a consumer is active in a failover subscription group
sijie commented on issue #1156: Introduce ActiveConsumerListener for realizing if a consumer is active in a failover subscription group URL: https://github.com/apache/incubator-pulsar/pull/1156#issuecomment-365652446 @merlimat as you suggested, I did two changes here: - rename `ActiveConsumerListener` to `ConsumerEventListener`. currently since we only use it for failover subscription, so I am still disallowing shared/exclusive subscription to use the listener. we can enable if we support more events in future. - I changed the dispatcher to use `consumer` object as the comparison identifier. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ivankelly commented on issue #1137: Schema registry
ivankelly commented on issue #1137: Schema registry URL: https://github.com/apache/incubator-pulsar/pull/1137#issuecomment-365634195 I don't see any client changes in this. How is a schema set for a topic? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] zhaijack commented on issue #1103: PIP-13-1/3: Provide `TopicsConsumer` to consume from several topics under same namespace
zhaijack commented on issue #1103: PIP-13-1/3: Provide `TopicsConsumer` to consume from several topics under same namespace URL: https://github.com/apache/incubator-pulsar/pull/1103#issuecomment-365620069 @merlimat @ivankelly, Thanks for your comments, Would you please help review it again? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] zhaijack commented on issue #1103: PIP-13-1/3: Provide `TopicsConsumer` to consume from several topics under same namespace
zhaijack commented on issue #1103: PIP-13-1/3: Provide `TopicsConsumer` to consume from several topics under same namespace URL: https://github.com/apache/incubator-pulsar/pull/1103#issuecomment-365613677 opened issue #1237 tracking using builder to create topics consumer. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] zhaijack commented on a change in pull request #1103: PIP-13-1/3: Provide `TopicsConsumer` to consume from several topics under same namespace
zhaijack commented on a change in pull request #1103: PIP-13-1/3: Provide `TopicsConsumer` to consume from several topics under same namespace URL: https://github.com/apache/incubator-pulsar/pull/1103#discussion_r168178825 ## File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/TopicsConsumerImpl.java ## @@ -0,0 +1,881 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pulsar.client.impl; + +import static com.google.common.base.Preconditions.checkArgument; +import static com.google.common.base.Preconditions.checkState; + +import com.google.common.collect.Lists; +import java.util.Collection; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Set; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ConcurrentLinkedQueue; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; +import java.util.concurrent.locks.ReadWriteLock; +import java.util.concurrent.locks.ReentrantReadWriteLock; +import java.util.stream.Collectors; +import java.util.stream.IntStream; + +import org.apache.pulsar.client.api.Consumer; +import org.apache.pulsar.client.api.ConsumerConfiguration; +import org.apache.pulsar.client.api.Message; +import org.apache.pulsar.client.api.MessageId; +import org.apache.pulsar.client.api.PulsarClientException; +import org.apache.pulsar.client.api.SubscriptionType; +import org.apache.pulsar.client.util.ConsumerName; +import org.apache.pulsar.client.util.FutureUtil; +import org.apache.pulsar.common.api.proto.PulsarApi.CommandAck.AckType; +import org.apache.pulsar.common.naming.DestinationName; +import org.apache.pulsar.common.naming.NamespaceName; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class TopicsConsumerImpl extends ConsumerBase { + +// All topics should be in same namespace +protected NamespaceName namespaceName; + +// Map, when get do ACK, consumer will by find by topic name +private final ConcurrentHashMap consumers; Review comment: Yes, the changes for TopicMessageIdImpl, TopicMessageImpl, and UnAckedMessageTracker is aimed to make PartitionedConsumerImpl and TopicsConsumerImpl could be easy merged in the future. opened issue #1236 tracking this. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] zhaijack opened a new issue #1236: Make TopicsConsumerImpl extends PartitionedConsumerImpl
zhaijack opened a new issue #1236: Make TopicsConsumerImpl extends PartitionedConsumerImpl URL: https://github.com/apache/incubator-pulsar/issues/1236 IN PIP-13 and its following PRs: https://github.com/apache/incubator-pulsar/pull/1103, We have made TopicsConsumerImpl behaviour similar to PartitionedConsumerImpl, This issue is tracking the following work to make TopicsConsumerImpl extends PartitionedConsumerImpl. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] zhaijack commented on a change in pull request #1103: PIP-13-1/3: Provide `TopicsConsumer` to consume from several topics under same namespace
zhaijack commented on a change in pull request #1103: PIP-13-1/3: Provide `TopicsConsumer` to consume from several topics under same namespace URL: https://github.com/apache/incubator-pulsar/pull/1103#discussion_r167877509 ## File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/TopicsConsumerImpl.java ## @@ -0,0 +1,881 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pulsar.client.impl; + +import static com.google.common.base.Preconditions.checkArgument; +import static com.google.common.base.Preconditions.checkState; + +import com.google.common.collect.Lists; +import java.util.Collection; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Set; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ConcurrentLinkedQueue; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; +import java.util.concurrent.locks.ReadWriteLock; +import java.util.concurrent.locks.ReentrantReadWriteLock; +import java.util.stream.Collectors; +import java.util.stream.IntStream; + +import org.apache.pulsar.client.api.Consumer; +import org.apache.pulsar.client.api.ConsumerConfiguration; +import org.apache.pulsar.client.api.Message; +import org.apache.pulsar.client.api.MessageId; +import org.apache.pulsar.client.api.PulsarClientException; +import org.apache.pulsar.client.api.SubscriptionType; +import org.apache.pulsar.client.util.ConsumerName; +import org.apache.pulsar.client.util.FutureUtil; +import org.apache.pulsar.common.api.proto.PulsarApi.CommandAck.AckType; +import org.apache.pulsar.common.naming.DestinationName; +import org.apache.pulsar.common.naming.NamespaceName; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class TopicsConsumerImpl extends ConsumerBase { + +// All topics should be in same namespace +protected NamespaceName namespaceName; + +// Map, when get do ACK, consumer will by find by topic name +private final ConcurrentHashMap consumers; Review comment: Thanks. Yes. There is a plan to make this extends PartitionedConsumerImpl, that is also the reason that this change keep use a lot of same interface. But it maybe good to make PartitionedConsumerImpl and TopicsConsumerImpl separate at first, this could avoid bring bugs into PartitionedConsumerImpl. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch asf-site updated: Updated site at revision 7404952
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git The following commit(s) were added to refs/heads/asf-site by this push: new 2716be2 Updated site at revision 7404952 2716be2 is described below commit 2716be27a9a9d3fd959055f19bda732a9e68879f Author: jenkinsAuthorDate: Wed Feb 14 08:16:04 2018 + Updated site at revision 7404952 --- content/api/client/index-all.html | 8 ++ .../org/apache/pulsar/client/api/Reader.html | 47 +- .../docs/latest/project/BinaryProtocol/index.html | 157 - content/ja/project/BinaryProtocol/index.html | 157 - 4 files changed, 358 insertions(+), 11 deletions(-) diff --git a/content/api/client/index-all.html b/content/api/client/index-all.html index 7a9aa8a..296466e 100644 --- a/content/api/client/index-all.html +++ b/content/api/client/index-all.html @@ -557,6 +557,14 @@ Check whether the message has a key +hasMessageAvailable() - Method in interface org.apache.pulsar.client.api.Reader + +Check if there is any message available to read from the current position. + +hasMessageAvailableAsync() - Method in interface org.apache.pulsar.client.api.Reader + +Asynchronously Check if there is message that has been published successfully to the broker in the topic. + hasProperty(String) - Method in interface org.apache.pulsar.client.api.Message Check whether the message has a specific property attached. diff --git a/content/api/client/org/apache/pulsar/client/api/Reader.html b/content/api/client/org/apache/pulsar/client/api/Reader.html index 052fce9..c8ae116 100644 --- a/content/api/client/org/apache/pulsar/client/api/Reader.html +++ b/content/api/client/org/apache/pulsar/client/api/Reader.html @@ -17,7 +17,7 @@ catch(err) { } //--> -var methods = {"i0":6,"i1":6,"i2":6,"i3":6,"i4":6,"i5":6}; +var methods = {"i0":6,"i1":6,"i2":6,"i3":6,"i4":6,"i5":6,"i6":6,"i7":6}; var tabs = {65535:["t0","All Methods"],2:["t2","Instance Methods"],4:["t3","Abstract Methods"]}; var altColor = "altColor"; var rowColor = "rowColor"; @@ -136,24 +136,36 @@ extends Closeable boolean +hasMessageAvailable() +Check if there is any message available to read from the current position. + + + +CompletableFutureBoolean +hasMessageAvailableAsync() +Asynchronously Check if there is message that has been published successfully to the broker in the topic. + + + +boolean hasReachedEndOfTopic() Return true if the topic was terminated and this reader has reached the end of the topic - + Message readNext() Read the next message in the topic - + Message readNext(inttimeout, TimeUnitunit) Read the next message in the topic. - + CompletableFutureMessage readNextAsync() @@ -253,13 +265,38 @@ extends Closeable - + hasReachedEndOfTopic booleanhasReachedEndOfTopic() Return true if the topic was terminated and this reader has reached the end of the topic + + + + + +hasMessageAvailable +booleanhasMessageAvailable() + throws PulsarClientException +Check if there is any message available to read from the current position. + +Throws: +PulsarClientException + + + + + + + + +hasMessageAvailableAsync +CompletableFutureBooleanhasMessageAvailableAsync() +Asynchronously Check if there is message that has been published successfully to the broker in the topic. + + diff --git a/content/docs/latest/project/BinaryProtocol/index.html b/content/docs/latest/project/BinaryProtocol/index.html index 29b81ec..d67dda9 100644 --- a/content/docs/latest/project/BinaryProtocol/index.html +++ b/content/docs/latest/project/BinaryProtocol/index.html @@ -1985,6 +1985,18 @@ response will contain actual metadata. + + CommandGetLastMessageId + + + + + + CommandGetLastMessageIdResponse + + + + CommandLookupTopic @@ -2505,6 +2517,26 @@ response will contain actual metadata. + + + getLastMessageId + + CommandGetLastMessageId + + optional + + + + + + getLastMessageIdResponse + + CommandGetLastMessageIdResponse + + optional + + + @@ -2794,7 +2826,7 @@ will be the auth of the proxy itself Original auth role and auth Method that was passed to the proxy. In this case the auth info above -will be the auth of the proxy itself +will be