merlimat closed pull request #1086: Pulsar proxy documentation URL: https://github.com/apache/incubator-pulsar/pull/1086
This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/site/Gemfile b/site/Gemfile index 242f76040..94e2a8722 100644 --- a/site/Gemfile +++ b/site/Gemfile @@ -20,4 +20,4 @@ source 'https://rubygems.org' ruby '2.4.1' -gem 'jekyll', '3.7.0' \ No newline at end of file +gem 'jekyll', '3.7.0' diff --git a/site/_data/cli/pulsar.yaml b/site/_data/cli/pulsar.yaml index 50d9488eb..f5ae17e77 100644 --- a/site/_data/cli/pulsar.yaml +++ b/site/_data/cli/pulsar.yaml @@ -79,6 +79,19 @@ commands: description: The web service URL for the new cluster with TLS encryption - flags: -zk, --zookeeper description: The local ZooKeeper quorum connection string +- name: proxy + description: Manages the Pulsar proxy + example: | + pulsar proxy \ + --zookeeper-servers zk-0,zk-1,zk2 \ + --global-zookeeper-servers zk-0,zk-1,zk-2 + options: + - flags: -c, --config + description: Path to a Pulsar proxy configuration file + - flags: -gzk, --global-zookeeper-servers + description: Global ZooKeeper connection string + - flags: -zk, --zookeeper-servers + description: Local ZooKeeper connection string - name: standalone description: Run a broker service with local bookies and local ZooKeeper options: diff --git a/site/_data/sidebar.yaml b/site/_data/sidebar.yaml index 4a543e2cb..c0264383b 100644 --- a/site/_data/sidebar.yaml +++ b/site/_data/sidebar.yaml @@ -65,6 +65,8 @@ groups: endpoint: Stats - title: Modular load manager endpoint: ModularLoadManager + - title: Pulsar proxy + endpoint: Proxy - title: Client libraries dir: clients diff --git a/site/docs/latest/admin/Proxy.md b/site/docs/latest/admin/Proxy.md new file mode 100644 index 000000000..07f1bc04d --- /dev/null +++ b/site/docs/latest/admin/Proxy.md @@ -0,0 +1,53 @@ +--- +title: The Pulsar proxy +--- + +<!-- + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +--> + +The [Pulsar proxy](../../getting-started/ConceptsAndArchitecture#pulsar-proxy) is an optional gateway that you can run over the {% popover brokers %} in a Pulsar {% popover cluster %}. We recommend running a Pulsar proxy in cases when direction connections between clients and Pulsar brokers are either infeasible, undesirable, or both, for example when running Pulsar in a cloud environment or on [Kubernetes](https://kubernetes.io) or an analogous platform. + +## Running the proxy + +In order to run the Pulsar proxy, you need to have both a local and global [ZooKeeper](https://zookeeper.apache.org) quorum set up for use by your Pulsar cluster. For instructions, see [this document](../../deployment/cluster). Once you have ZooKeeper set up and have connection strings for both ZooKeeper quorums, you can use the [`proxy`](../../reference/CliTools#pulsar-proxy) command of the [`pulsar`](../../reference/CliTools#pulsar) CLI tool to start up the proxy (preferably on its own machine or in its own VM): + +To start the proxy: + +```bash +$ cd /path/to/pulsar/directory +$ bin/pulsar proxy \ + --zookeeper-servers zk-0,zk-1,zk-2 \ + --global-zookeeper-servers zk-0,zk-1,zk-2 +``` + +{% include admonition.html type="success" content="You can run as many instances of the Pulsar proxy in a cluster as you would like." %} + +## Stopping the proxy + +The Pulsar proxy runs by default in the foreground. To stop the proxy, simply stop the process in which it's running. + +## Proxy frontends + +We recommend running the Pulsar proxy behind some kind of load-distributing frontend, such as an [HAProxy](https://www.digitalocean.com/community/tutorials/an-introduction-to-haproxy-and-load-balancing-concepts) load balancer. + +## Using Pulsar clients with the proxy + +Once your Pulsar proxy is up and running, preferably behind a load-distributing [frontend](#proxy-frontends), clients can connect to the proxy via whichever address is used by the frontend. If the address were the DNS address `pulsar.cluster.default`, for example, then the connection URL for clients would be `pulsar://pulsar.cluster.default:6650`. \ No newline at end of file diff --git a/site/docs/latest/getting-started/ConceptsAndArchitecture.md b/site/docs/latest/getting-started/ConceptsAndArchitecture.md index 22a04402d..f69fd7867 100644 --- a/site/docs/latest/getting-started/ConceptsAndArchitecture.md +++ b/site/docs/latest/getting-started/ConceptsAndArchitecture.md @@ -307,6 +307,31 @@ When an application wants to create a producer/consumer, the Pulsar client libra Whenever the TCP connection breaks, the client will immediately re-initiate this setup phase and will keep trying with exponential backoff to re-establish the producer or consumer until the operation succeeds. +## Pulsar proxy + +One way for Pulsar clients to interact with a Pulsar [cluster](#clusters) is by connecting to Pulsar message [brokers](#brokers) directly. In some cases, however, this kind of direct connection is either infeasible or undesirable because the client doesn't have direct access to broker addresses. If you're running Pulsar in a cloud environment or on [Kubernetes](https://kubernetes.io) or an analogous platform, for example, then direct client connections to brokers are likely not possible. + +The **Pulsar proxy** provides a solution to this problem by acting as a single gateway for all of the brokers in a cluster. If you run the Pulsar proxy (which, again, is optional), all client connections with the Pulsar {% popover cluster %} will flow through the proxy rather than communicating with brokers. + +{% include admonition.html type="success" content="For the sake of performance and fault tolerance, you can run as many instances of the Pulsar proxy as you'd like." %} + +Architecturally, the Pulsar proxy gets all the information it requires from ZooKeeper. When starting the proxy on a machine, you only need to provide ZooKeeper connection strings for the cluster-specific and {% popover global ZooKeeper %} clusters. Here's an example: + +```bash +$ bin/pulsar proxy \ + --zookeeper-servers zk-0,zk-1,zk-2 \ + --global-zookeeper-servers zk-0,zk-1,zk-2 +``` + +{% include admonition.html type="info" title="Pulsar proxy docs" content=' +For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](../../admin/Proxy). +' %} + +Some important things to know about the Pulsar proxy: + +* Connecting clients don't need to provide *any* specific configuration to use the Pulsar proxy. You won't need to update the client configuration for existing applications beyond updating the IP used for the service URL (for example if you're running a load balancer over the Pulsar proxy). +* [TLS encryption and authentication](../../admin/Authz/#tls-client-auth) is supported by the Pulsar proxy + ## Service discovery [Clients](../../getting-started/Clients) connecting to Pulsar {% popover brokers %} need to be able to communicate with an entire Pulsar {% popover instance %} using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](../../deployment/InstanceSetup#service-discovery-setup) guide. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
