This is an automated email from the ASF dual-hosted git repository.
sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new c4ae6c5 Using Pulsar as a message queue (cookbook) (#1507)
c4ae6c5 is described below
commit c4ae6c53ec7dfea1cd6cdbc891188473e4f196e4
Author: Luc Perkins <lucperk...@gmail.com>
AuthorDate: Thu Apr 26 13:42:45 2018 -0700
Using Pulsar as a message queue (cookbook) (#1507)
* change directory structure to include cookbooks
* add new message queue cookbook
* Finish first draft of doc
* add tags to new doc
* add missing subscription name to java example
* Add C++ example
* Add subscription name setting to Java example
* change section on receiver queue
---
site/_data/sidebar.yaml | 2 +
site/docs/latest/cookbooks/Encryption.md | 1 +
site/docs/latest/cookbooks/PartitionedTopics.md | 1 +
site/docs/latest/cookbooks/RetentionExpiry.md | 2 +-
site/docs/latest/cookbooks/message-queue.md | 113 ++++++++++++++++++++++++
5 files changed, 118 insertions(+), 1 deletion(-)
diff --git a/site/_data/sidebar.yaml b/site/_data/sidebar.yaml
index 68fe426..a553bee 100644
--- a/site/_data/sidebar.yaml
+++ b/site/_data/sidebar.yaml
@@ -142,6 +142,8 @@ groups:
endpoint: RetentionExpiry
- title: Encryption
endpoint: Encryption
+ - title: Using Pulsar as a message queue
+ endpoint: message-queue
- title: Developing Pulsar
dir: project
diff --git a/site/docs/latest/cookbooks/Encryption.md
b/site/docs/latest/cookbooks/Encryption.md
index 5ef80c1..942ce1f 100644
--- a/site/docs/latest/cookbooks/Encryption.md
+++ b/site/docs/latest/cookbooks/Encryption.md
@@ -5,6 +5,7 @@ tags:
- crypto
- encryption
- clients
+- cookbook
---
<!--
diff --git a/site/docs/latest/cookbooks/PartitionedTopics.md
b/site/docs/latest/cookbooks/PartitionedTopics.md
index aa6004d..deb4954 100644
--- a/site/docs/latest/cookbooks/PartitionedTopics.md
+++ b/site/docs/latest/cookbooks/PartitionedTopics.md
@@ -6,6 +6,7 @@ tags:
- partitioning
- admin
- clients
+- cookbook
---
<!--
diff --git a/site/docs/latest/cookbooks/RetentionExpiry.md
b/site/docs/latest/cookbooks/RetentionExpiry.md
index 68448c0..8191e56 100644
--- a/site/docs/latest/cookbooks/RetentionExpiry.md
+++ b/site/docs/latest/cookbooks/RetentionExpiry.md
@@ -1,7 +1,7 @@
---
title: Message retention and expiry
lead: Manage how long messages remain stored in your Pulsar instance
-tags: [admin, expiry, retention, backlog]
+tags: [admin, expiry, retention, backlog, cookbook]
---
<!--
diff --git a/site/docs/latest/cookbooks/message-queue.md
b/site/docs/latest/cookbooks/message-queue.md
new file mode 100644
index 0000000..dd2a77d
--- /dev/null
+++ b/site/docs/latest/cookbooks/message-queue.md
@@ -0,0 +1,113 @@
+---
+title: Using Pulsar as a message queue
+lead: Although Pulsar is typically known as a real-time messaging system, it's
also an excellent choice for a queuing system
+tags: [clients, java, python, message queue, cookbook]
+---
+
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+
+Message queues are essential components of many large-scale data
architectures. If every single work object that passes through your system
absolutely *must* be processed in spite of the slowness or downright failure of
this or that system component, there's a good chance that you'll need a message
queue to step in and ensure that unprocessed data is retained---with correct
ordering---until the required actions are taken.
+
+Pulsar is a great choice for a message queue because:
+
+* it was built with [persistent message
storage](../../getting-started/ConceptsAndArchitecture#persistent-storage) in
mind
+* it offers automatic load balancing across {% popover consumers %} for
messages on a topic (or custom load balancing if you wish)
+
+{% include admonition.html type="success" content="You can use the same Pulsar
installation to act as a real-time message bus and as a message queue if you
wish (or just one or the other). You can set aside some topics for real-time
purposes and other topics for message queue purposes (or use specific
namespaces for either purpose if you wish)." %}
+
+## Client configuration changes
+
+To use a Pulsar {% popover topic %} as a message queue, you should distribute
the receiver load on that topic across several {% popover consumers %} (the
optimal number of consumers will depend on the load). Each consumer must:
+
+* Establish a [shared
subscription](../../getting-started/ConceptsAndArchitecture#shared) and use the
same subscription name as the other consumers (otherwise the subscription is
not shared and the consumers can't act as a processing ensemble)
+* If you'd like to have tight control over message dispatching across
consumers, set the consumers' **receiver queue** size very low (potentially
even to 0 if necessary). Each Pulsar {% popover consumer %} has a receiver
queue that determines how many messages the consumer will attempt to fetch at a
time. A receiver queue of 1000 (the default), for example, means that the
consumer will attempt to process 1000 messages from the topic's backlog upon
connection. Setting the receiver queue t [...]
+
+ The downside to restricting the receiver queue size of consumers is that
that limits the potential throughput of those consumers and cannot be used with
{% popover partitioned topics %}. Whether the performance/control trade-off is
worthwhile will depend on your use case.
+
+### Java clients {#java}
+
+Here's an example Java consumer configuration that uses a shared subscription:
+
+```java
+import org.apache.pulsar.client.api.Consumer;
+import org.apache.pulsar.client.api.PulsarClient;
+import org.apache.pulsar.client.api.SubscriptionType;
+
+String SERVICE_URL = "pulsar://localhost:6650";
+String TOPIC = "persistent://sample/standalone/ns1/mq-topic-1";
+String subscription = "sub-1";
+
+PulsarClient client = PulsarClient.builder()
+ .serviceUrl(SERVICE_URL)
+ .build();
+
+Consumer consumer = client.newConsumer()
+ .topic(TOPIC)
+ .subscriptionName(subscription)
+ .subscriptionType(SubscriptionType.Shared)
+ // If you'd like to restrict the receiver queue size
+ .receiverQueueSize(10)
+ .subscribe();
+```
+
+### Python clients {#python}
+
+Here's an example Python consumer configuration that uses a shared
subscription:
+
+```python
+from pulsar import Client, ConsumerType
+
+SERVICE_URL = "pulsar://localhost:6650"
+TOPIC = "persistent://sample/standalone/ns1/mq-topic-1"
+SUBSCRIPTION = "sub-1"
+
+client = Client(SERVICE_URL)
+consumer = client.subscribe(
+ TOPIC,
+ SUBSCRIPTION,
+ # If you'd like to restrict the receiver queue size
+ receiver_queue_size=10,
+ consumer_type=ConsumerType.Shared)
+```
+
+### C++ clients {#cpp}
+
+Here's an example C++ consumer configuration that uses a shared subscription:
+
+```cpp
+#include <pulsar/Client.h>
+
+std::string serviceUrl = "pulsar://localhost:6650";
+std::string topic = "persistent://sample/standalone/ns1/mq-topic-1";
+std::string subscription = "sub-1";
+
+Client client(serviceUrl);
+
+ConsumerConfiguration consumerConfig;
+consumerConfig.setConsumerType(ConsumerType.ConsumerShared);
+// If you'd like to restrict the receiver queue size
+consumerConfig.setReceiverQueueSize(10);
+
+Consumer consumer;
+
+Result result =
client.subscribe("persistent://sample/standalone/ns1/my-topic", subscription,
consumerConfig, consumer);
+```
\ No newline at end of file
--
To stop receiving notification emails like this one, please contact
si...@apache.org.
