This is an automated email from the ASF dual-hosted git repository. sijie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/master by this push: new c4ae6c5 Using Pulsar as a message queue (cookbook) (#1507) c4ae6c5 is described below commit c4ae6c53ec7dfea1cd6cdbc891188473e4f196e4 Author: Luc Perkins <lucperk...@gmail.com> AuthorDate: Thu Apr 26 13:42:45 2018 -0700 Using Pulsar as a message queue (cookbook) (#1507) * change directory structure to include cookbooks * add new message queue cookbook * Finish first draft of doc * add tags to new doc * add missing subscription name to java example * Add C++ example * Add subscription name setting to Java example * change section on receiver queue --- site/_data/sidebar.yaml | 2 + site/docs/latest/cookbooks/Encryption.md | 1 + site/docs/latest/cookbooks/PartitionedTopics.md | 1 + site/docs/latest/cookbooks/RetentionExpiry.md | 2 +- site/docs/latest/cookbooks/message-queue.md | 113 ++++++++++++++++++++++++ 5 files changed, 118 insertions(+), 1 deletion(-) diff --git a/site/_data/sidebar.yaml b/site/_data/sidebar.yaml index 68fe426..a553bee 100644 --- a/site/_data/sidebar.yaml +++ b/site/_data/sidebar.yaml @@ -142,6 +142,8 @@ groups: endpoint: RetentionExpiry - title: Encryption endpoint: Encryption + - title: Using Pulsar as a message queue + endpoint: message-queue - title: Developing Pulsar dir: project diff --git a/site/docs/latest/cookbooks/Encryption.md b/site/docs/latest/cookbooks/Encryption.md index 5ef80c1..942ce1f 100644 --- a/site/docs/latest/cookbooks/Encryption.md +++ b/site/docs/latest/cookbooks/Encryption.md @@ -5,6 +5,7 @@ tags: - crypto - encryption - clients +- cookbook --- <!-- diff --git a/site/docs/latest/cookbooks/PartitionedTopics.md b/site/docs/latest/cookbooks/PartitionedTopics.md index aa6004d..deb4954 100644 --- a/site/docs/latest/cookbooks/PartitionedTopics.md +++ b/site/docs/latest/cookbooks/PartitionedTopics.md @@ -6,6 +6,7 @@ tags: - partitioning - admin - clients +- cookbook --- <!-- diff --git a/site/docs/latest/cookbooks/RetentionExpiry.md b/site/docs/latest/cookbooks/RetentionExpiry.md index 68448c0..8191e56 100644 --- a/site/docs/latest/cookbooks/RetentionExpiry.md +++ b/site/docs/latest/cookbooks/RetentionExpiry.md @@ -1,7 +1,7 @@ --- title: Message retention and expiry lead: Manage how long messages remain stored in your Pulsar instance -tags: [admin, expiry, retention, backlog] +tags: [admin, expiry, retention, backlog, cookbook] --- <!-- diff --git a/site/docs/latest/cookbooks/message-queue.md b/site/docs/latest/cookbooks/message-queue.md new file mode 100644 index 0000000..dd2a77d --- /dev/null +++ b/site/docs/latest/cookbooks/message-queue.md @@ -0,0 +1,113 @@ +--- +title: Using Pulsar as a message queue +lead: Although Pulsar is typically known as a real-time messaging system, it's also an excellent choice for a queuing system +tags: [clients, java, python, message queue, cookbook] +--- + +<!-- + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +--> + +Message queues are essential components of many large-scale data architectures. If every single work object that passes through your system absolutely *must* be processed in spite of the slowness or downright failure of this or that system component, there's a good chance that you'll need a message queue to step in and ensure that unprocessed data is retained---with correct ordering---until the required actions are taken. + +Pulsar is a great choice for a message queue because: + +* it was built with [persistent message storage](../../getting-started/ConceptsAndArchitecture#persistent-storage) in mind +* it offers automatic load balancing across {% popover consumers %} for messages on a topic (or custom load balancing if you wish) + +{% include admonition.html type="success" content="You can use the same Pulsar installation to act as a real-time message bus and as a message queue if you wish (or just one or the other). You can set aside some topics for real-time purposes and other topics for message queue purposes (or use specific namespaces for either purpose if you wish)." %} + +## Client configuration changes + +To use a Pulsar {% popover topic %} as a message queue, you should distribute the receiver load on that topic across several {% popover consumers %} (the optimal number of consumers will depend on the load). Each consumer must: + +* Establish a [shared subscription](../../getting-started/ConceptsAndArchitecture#shared) and use the same subscription name as the other consumers (otherwise the subscription is not shared and the consumers can't act as a processing ensemble) +* If you'd like to have tight control over message dispatching across consumers, set the consumers' **receiver queue** size very low (potentially even to 0 if necessary). Each Pulsar {% popover consumer %} has a receiver queue that determines how many messages the consumer will attempt to fetch at a time. A receiver queue of 1000 (the default), for example, means that the consumer will attempt to process 1000 messages from the topic's backlog upon connection. Setting the receiver queue t [...] + + The downside to restricting the receiver queue size of consumers is that that limits the potential throughput of those consumers and cannot be used with {% popover partitioned topics %}. Whether the performance/control trade-off is worthwhile will depend on your use case. + +### Java clients {#java} + +Here's an example Java consumer configuration that uses a shared subscription: + +```java +import org.apache.pulsar.client.api.Consumer; +import org.apache.pulsar.client.api.PulsarClient; +import org.apache.pulsar.client.api.SubscriptionType; + +String SERVICE_URL = "pulsar://localhost:6650"; +String TOPIC = "persistent://sample/standalone/ns1/mq-topic-1"; +String subscription = "sub-1"; + +PulsarClient client = PulsarClient.builder() + .serviceUrl(SERVICE_URL) + .build(); + +Consumer consumer = client.newConsumer() + .topic(TOPIC) + .subscriptionName(subscription) + .subscriptionType(SubscriptionType.Shared) + // If you'd like to restrict the receiver queue size + .receiverQueueSize(10) + .subscribe(); +``` + +### Python clients {#python} + +Here's an example Python consumer configuration that uses a shared subscription: + +```python +from pulsar import Client, ConsumerType + +SERVICE_URL = "pulsar://localhost:6650" +TOPIC = "persistent://sample/standalone/ns1/mq-topic-1" +SUBSCRIPTION = "sub-1" + +client = Client(SERVICE_URL) +consumer = client.subscribe( + TOPIC, + SUBSCRIPTION, + # If you'd like to restrict the receiver queue size + receiver_queue_size=10, + consumer_type=ConsumerType.Shared) +``` + +### C++ clients {#cpp} + +Here's an example C++ consumer configuration that uses a shared subscription: + +```cpp +#include <pulsar/Client.h> + +std::string serviceUrl = "pulsar://localhost:6650"; +std::string topic = "persistent://sample/standalone/ns1/mq-topic-1"; +std::string subscription = "sub-1"; + +Client client(serviceUrl); + +ConsumerConfiguration consumerConfig; +consumerConfig.setConsumerType(ConsumerType.ConsumerShared); +// If you'd like to restrict the receiver queue size +consumerConfig.setReceiverQueueSize(10); + +Consumer consumer; + +Result result = client.subscribe("persistent://sample/standalone/ns1/my-topic", subscription, consumerConfig, consumer); +``` \ No newline at end of file -- To stop receiving notification emails like this one, please contact si...@apache.org.