This is an automated email from the ASF dual-hosted git repository.
urfree pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/pulsar-site.git
The following commit(s) were added to refs/heads/main by this push:
new 392ea69eec9 Docs sync done from apache/pulsar (#ae98392)
392ea69eec9 is described below
commit 392ea69eec95ce66199c739ff140e47a8c72c3a9
Author: github-actions[bot]
<41898282+github-actions[bot]@users.noreply.github.com>
AuthorDate: Thu Dec 29 06:15:30 2022 +0000
Docs sync done from apache/pulsar (#ae98392)
---
docs/client-libraries-java.md | 2 +-
docs/client-libraries.md | 7 ++---
docs/io-aerospike-sink.md | 6 ++++
docs/io-alluxio.md | 6 ++++
docs/io-canal-source.md | 24 ++++++++++------
docs/io-cassandra-sink.md | 6 ++++
docs/io-cdc-debezium.md | 6 ++++
docs/io-connectors.md | 8 +++++-
docs/io-debezium-source.md | 65 +++++++++++++++++++++++--------------------
docs/io-debug.md | 43 +++++++++++++++++++---------
docs/io-dynamodb-source.md | 6 ++++
docs/io-elasticsearch-sink.md | 6 ++++
docs/io-file-source.md | 16 +++++++----
docs/io-flume-sink.md | 6 ++++
docs/io-flume-source.md | 6 ++++
docs/io-hbase-sink.md | 6 ++++
docs/io-hdfs2-sink.md | 6 ++++
docs/io-hdfs3-sink.md | 6 ++++
docs/io-http-sink.md | 6 ++++
docs/io-influxdb-sink.md | 6 ++++
docs/io-jdbc-sink.md | 6 ++++
docs/io-kafka-sink.md | 6 ++++
docs/io-kafka-source.md | 36 ++++++++++++++----------
docs/io-kinesis-sink.md | 6 ++++
docs/io-kinesis-source.md | 6 ++++
docs/io-mongo-sink.md | 6 ++++
docs/io-mongo-source.md | 6 ++++
docs/io-netty-source.md | 6 ++++
docs/io-nsq-source.md | 6 ++++
docs/io-rabbitmq-sink.md | 6 ++++
docs/io-rabbitmq-source.md | 6 ++++
docs/io-redis-sink.md | 30 +++++++++++---------
docs/io-solr-sink.md | 21 ++++++++------
docs/io-twitter-source.md | 6 ++++
34 files changed, 296 insertions(+), 100 deletions(-)
diff --git a/docs/client-libraries-java.md b/docs/client-libraries-java.md
index 998649f4a8f..dc22cd41889 100644
--- a/docs/client-libraries-java.md
+++ b/docs/client-libraries-java.md
@@ -217,7 +217,7 @@ Name| Type | <div>Description</div>| Default
`sendTimeoutMs`| long|Message send timeout in ms.<br />If a message is not
acknowledged by a server before the `sendTimeout` expires, an error
occurs.|30000
`blockIfQueueFull`|boolean|If it is set to `true`, when the outgoing message
queue is full, the `Send` and `SendAsync` methods of producer block, rather
than failing and throwing errors. <br />If it is set to `false`, when the
outgoing message queue is full, the `Send` and `SendAsync` methods of producer
fail and `ProducerQueueIsFullError` exceptions occur.<br /><br />The
`MaxPendingMessages` parameter determines the size of the outgoing message
queue.|false
`maxPendingMessages`| int|The maximum size of a queue holding pending
messages.<br /><br />For example, a message waiting to receive an
acknowledgment from a [broker](reference-terminology.md#broker). <br /><br />By
default, when the queue is full, all calls to the `Send` and `SendAsync`
methods fail **unless** you set `BlockIfQueueFull` to `true`.|1000
-`maxPendingMessagesAcrossPartitions`|int|Deprecated, use `memoryLimit`
instead. The maximum number of pending messages across partitions. <br /><br
/>Use the setting to lower the max pending messages for each partition ({@link
#setMaxPendingMessages(int)}) if the total number exceeds the configured value
and maxPendingMessagesAcrossPartitions needs to be >= maxPendingMessages.|50000
+`maxPendingMessagesAcrossPartitions`|int|Deprecated, use `memoryLimit`
instead. The maximum number of pending messages across partitions. <br /><br
/>Use the setting to lower the max pending messages for each partition ({@link
setMaxPendingMessages(int)}) if the total number exceeds the configured value
and maxPendingMessagesAcrossPartitions needs to be >= maxPendingMessages.|50000
`messageRoutingMode`| MessageRoutingMode|Message routing logic for producers
on [partitioned
topics](concepts-architecture-overview.md#partitioned-topics).<br /> Apply the
logic only when setting no key on messages. <br />Available options are as
follows: <br /><li>`pulsar.RoundRobinDistribution`: round
robin</li><li>`pulsar.UseSinglePartition`: publish all messages to a single
partition</li><li>`pulsar.CustomPartition`: a custom partitioning
scheme</li>|<li>`pulsar.RoundRobinDistribution`</li>
`hashingScheme`| HashingScheme|Hashing function determining the partition
where you publish a particular message (**partitioned topics only**).<br
/>Available options are as follows:<br /><li> `pulsar.JavastringHash`: the
equivalent of `string.hashCode()` in Java</li><li> `pulsar.Murmur3_32Hash`:
applies the [Murmur3](https://en.wikipedia.org/wiki/MurmurHash) hashing
function</li><li>`pulsar.BoostHash`: applies the hashing function from C++'s
[Boost](https://www.boost.org/doc/libs/1_62_0 [...]
`cryptoFailureAction`| ProducerCryptoFailureAction|Producer should take action
when encryption fails.<br /><li>**FAIL**: if encryption fails, unencrypted
messages fail to send.</li><li> **SEND**: if encryption fails, unencrypted
messages are sent.</li> |`ProducerCryptoFailureAction.FAIL`
diff --git a/docs/client-libraries.md b/docs/client-libraries.md
index d47951f8643..e925bb57065 100644
--- a/docs/client-libraries.md
+++ b/docs/client-libraries.md
@@ -40,10 +40,9 @@ Besides the officially released clients, multiple projects
on developing Pulsar
### Go
-| Project |
Description | License
| Badges
|
-| --------------------------------------------------------------- |
---------------------------------------- |
--------------------------------------------------------- |
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
-| [pulsar-client-go](https://github.com/Comcast/pulsar-client-go) | A Go
client library for Apache Pulsar |
[Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0) | 
|
-| [go-pulsar](https://github.com/t2y/go-pulsar) | go-pulsar
is a client library for Pulsar |
[Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0) | 
|
+| Project |
Description | License
| Badges
|
+| --------------------------------------------------------------- |
---------------------------------------- |
---------------------------------------------------------
|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| [pulsar-client-go](https://github.com/Comcast/pulsar-client-go) | A Go
client library for Apache Pulsar |
[Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0) | 
|
### Haskell
diff --git a/docs/io-aerospike-sink.md b/docs/io-aerospike-sink.md
index 8e423de9813..6d2986fbf50 100644
--- a/docs/io-aerospike-sink.md
+++ b/docs/io-aerospike-sink.md
@@ -4,6 +4,12 @@ title: Aerospike sink connector
sidebar_label: "Aerospike sink connector"
---
+:::note
+
+You can download all the Pulsar connectors on [download
page](https://pulsar.apache.org/download).
+
+::::
+
The Aerospike sink connector pulls messages from Pulsar topics to Aerospike
clusters.
## Configuration
diff --git a/docs/io-alluxio.md b/docs/io-alluxio.md
index 59c3247f9e6..80253a5332a 100644
--- a/docs/io-alluxio.md
+++ b/docs/io-alluxio.md
@@ -4,6 +4,12 @@ title: Alluxio sink connector
sidebar_label: Alluxio sink connector
---
+:::note
+
+You can download all the Pulsar connectors on [download
page](https://pulsar.apache.org/download).
+
+::::
+
## Sink
The Alluxio sink connector pulls messages from Pulsar topics and persists the
messages to an Alluxio directory.
diff --git a/docs/io-canal-source.md b/docs/io-canal-source.md
index 724cd5b1504..9cbbc037c4e 100644
--- a/docs/io-canal-source.md
+++ b/docs/io-canal-source.md
@@ -4,6 +4,12 @@ title: Canal source connector
sidebar_label: "Canal source connector"
---
+:::note
+
+You can download all the Pulsar connectors on [download
page](https://pulsar.apache.org/download).
+
+::::
+
The Canal source connector pulls messages from MySQL to Pulsar topics.
## Configuration
@@ -71,7 +77,7 @@ Here is an example of storing MySQL data using the
configuration file as above.
2. Create a configuration file `mysqld.cnf`.
- ```bash
+ ```properties
[mysqld]
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
@@ -163,14 +169,14 @@ Here is an example of storing MySQL data using the
configuration file as above.
docker exec -it pulsar-standalone /bin/bash
wget
https://archive.apache.org/dist/pulsar/pulsar-2.3.0/connectors/pulsar-io-canal-2.3.0.nar
-P connectors
./bin/pulsar-admin source localrun \
- --archive ./connectors/pulsar-io-canal-2.3.0.nar \
- --classname org.apache.pulsar.io.canal.CanalStringSource \
- --tenant public \
- --namespace default \
- --name canal \
- --destination-topic-name my-topic \
- --source-config-file /pulsar/conf/canal-mysql-source-config.yaml \
- --parallelism 1
+ --archive ./connectors/pulsar-io-canal-2.3.0.nar \
+ --classname org.apache.pulsar.io.canal.CanalStringSource \
+ --tenant public \
+ --namespace default \
+ --name canal \
+ --destination-topic-name my-topic \
+ --source-config-file /pulsar/conf/canal-mysql-source-config.yaml \
+ --parallelism 1
```
12. Consume data from MySQL.
diff --git a/docs/io-cassandra-sink.md b/docs/io-cassandra-sink.md
index 59eafdf4cf8..b5de5f58a79 100644
--- a/docs/io-cassandra-sink.md
+++ b/docs/io-cassandra-sink.md
@@ -4,6 +4,12 @@ title: Cassandra sink connector
sidebar_label: "Cassandra sink connector"
---
+:::note
+
+You can download all the Pulsar connectors on [download
page](https://pulsar.apache.org/download).
+
+::::
+
The Cassandra sink connector pulls messages from Pulsar topics to Cassandra
clusters.
## Configuration
diff --git a/docs/io-cdc-debezium.md b/docs/io-cdc-debezium.md
index 981fe1926ec..70010e478a3 100644
--- a/docs/io-cdc-debezium.md
+++ b/docs/io-cdc-debezium.md
@@ -4,6 +4,12 @@ title: Debezium source connector
sidebar_label: "Debezium source connector"
---
+:::note
+
+You can download all the Pulsar connectors on [download
page](https://pulsar.apache.org/download).
+
+::::
+
The Debezium source connector pulls messages from MySQL or PostgreSQL and
persists the messages to Pulsar topics.
## Configuration
diff --git a/docs/io-connectors.md b/docs/io-connectors.md
index 362c150e752..1f88d40f67a 100644
--- a/docs/io-connectors.md
+++ b/docs/io-connectors.md
@@ -139,7 +139,13 @@ Pulsar has various sink connectors, which are sorted
alphabetically as below.
* [Configuration](io-aerospike-sink.md#configuration)
* [Java
class](https://github.com/apache/pulsar/blob/master/pulsar-io/aerospike/src/main/java/org/apache/pulsar/io/aerospike/AerospikeStringSink.java)
-
+
+### Alluxio
+
+* [Configuration](io-alluxio.md#configuration)
+
+* [Java
class](https://github.com/apache/pulsar/blob/master/pulsar-io/alluxio/src/main/java/org/apache/pulsar/io/alluxio/sink/AlluxioSink.java)
+
### Cassandra
* [Configuration](io-cassandra-sink.md#configuration)
diff --git a/docs/io-debezium-source.md b/docs/io-debezium-source.md
index 0095f006e2f..ff2ad926ffe 100644
--- a/docs/io-debezium-source.md
+++ b/docs/io-debezium-source.md
@@ -4,6 +4,12 @@ title: Debezium source connector
sidebar_label: "Debezium source connector"
---
+:::note
+
+You can download all the Pulsar connectors on [download
page](https://pulsar.apache.org/download).
+
+::::
+
The Debezium source connector pulls messages from MySQL or PostgreSQL and
persists the messages to Pulsar topics.
## Configuration
@@ -169,11 +175,11 @@ This example shows how to change the data of a MySQL
table using the Pulsar Debe
```bash
bin/pulsar-admin source localrun \
- --archive connectors/pulsar-io-debezium-mysql-@pulsar:[email protected] \
- --name debezium-mysql-source \
- --tenant public \
- --namespace default \
- --source-config '{"database.hostname": "localhost","database.port":
"3306","database.user": "debezium","database.password":
"dbz","database.server.id": "184054","database.server.name":
"dbserver1","database.whitelist": "inventory","database.history":
"org.apache.pulsar.io.debezium.PulsarDatabaseHistory","database.history.pulsar.topic":
"history-topic","database.history.pulsar.service.url":
"pulsar://127.0.0.1:6650","key.converter":
"org.apache.kafka.connect.json.JsonConverter","va [...]
+ --archive connectors/pulsar-io-debezium-mysql-@pulsar:[email protected] \
+ --name debezium-mysql-source \
+ --tenant public \
+ --namespace default \
+ --source-config '{"database.hostname": "localhost","database.port":
"3306","database.user": "debezium","database.password":
"dbz","database.server.id": "184054","database.server.name":
"dbserver1","database.whitelist": "inventory","database.history":
"org.apache.pulsar.io.debezium.PulsarDatabaseHistory","database.history.pulsar.topic":
"history-topic","database.history.pulsar.service.url":
"pulsar://127.0.0.1:6650","key.converter":
"org.apache.kafka.connect.json.JsonConverter" [...]
```
:::note
@@ -193,7 +199,7 @@ This example shows how to change the data of a MySQL table
using the Pulsar Debe
```bash
bin/pulsar-admin source localrun \
- --source-config-file debezium-mysql-source-config.yaml
+ --source-config-file debezium-mysql-source-config.yaml
```
4. Subscribe to the topic _sub-products_ for the table _inventory.products_.
@@ -204,13 +210,13 @@ This example shows how to change the data of a MySQL
table using the Pulsar Debe
5. Start a MySQL client in docker.
- ```bash
- docker run -it --rm \
- --name mysqlterm \
- --link mysql \
- --rm mysql:5.7 sh \
- -c 'exec mysql -h"$MYSQL_PORT_3306_TCP_ADDR" -P"$MYSQL_PORT_3306_TCP_PORT"
-uroot -p"$MYSQL_ENV_MYSQL_ROOT_PASSWORD"'
- ```
+ ```bash
+ docker run -it --rm \
+ --name mysqlterm \
+ --link mysql \
+ --rm mysql:5.7 sh \
+ -c 'exec mysql -h"$MYSQL_PORT_3306_TCP_ADDR"
-P"$MYSQL_PORT_3306_TCP_PORT" -uroot -p"$MYSQL_ENV_MYSQL_ROOT_PASSWORD"'
+ ```
6. A MySQL client pops out.
@@ -264,7 +270,6 @@ You can use one of the following methods to create a
configuration file.
parallelism: 1
configs:
-
## config for postgres version 10+, official docker image: postgres:<10+>
database.hostname: "localhost"
database.port: "5432"
@@ -291,10 +296,10 @@ This example shows how to change the data of a PostgreSQL
table using the Pulsar
```bash
docker run -d -it --rm \
- --name pulsar-postgres \
- -p 5432:5432 \
- -e POSTGRES_PASSWORD=changeme \
- postgres:13.3 -c wal_level=logical
+ --name pulsar-postgres \
+ -p 5432:5432 \
+ -e POSTGRES_PASSWORD=changeme \
+ postgres:13.3 -c wal_level=logical
```
2. Start a Pulsar service locally in standalone mode.
@@ -311,11 +316,11 @@ This example shows how to change the data of a PostgreSQL
table using the Pulsar
```bash
bin/pulsar-admin source localrun \
- --archive connectors/pulsar-io-debezium-postgres-@pulsar:[email protected] \
- --name debezium-postgres-source \
- --tenant public \
- --namespace default \
- --source-config '{"database.hostname": "localhost","database.port":
"5432","database.user": "postgres","database.password":
"changeme","database.dbname": "postgres","database.server.name":
"dbserver1","schema.whitelist": "public","table.whitelist":
"public.users","pulsar.service.url": "pulsar://127.0.0.1:6650"}'
+ --archive
connectors/pulsar-io-debezium-postgres-@pulsar:[email protected] \
+ --name debezium-postgres-source \
+ --tenant public \
+ --namespace default \
+ --source-config '{"database.hostname": "localhost","database.port":
"5432","database.user": "postgres","database.password":
"changeme","database.dbname": "postgres","database.server.name":
"dbserver1","schema.whitelist": "public","table.whitelist":
"public.users","pulsar.service.url": "pulsar://127.0.0.1:6650"}'
```
:::note
@@ -335,7 +340,7 @@ This example shows how to change the data of a PostgreSQL
table using the Pulsar
```bash
bin/pulsar-admin source localrun \
- --source-config-file debezium-postgres-source-config.yaml
+ --source-config-file debezium-postgres-source-config.yaml
```
4. Subscribe to the topic _sub-users_ for the _public.users_ table.
@@ -471,11 +476,11 @@ This example shows how to change the data of a MongoDB
table using the Pulsar De
```bash
bin/pulsar-admin source localrun \
- --archive connectors/pulsar-io-debezium-mongodb-@pulsar:[email protected] \
- --name debezium-mongodb-source \
- --tenant public \
- --namespace default \
- --source-config '{"mongodb.hosts": "rs0/mongodb:27017","mongodb.name":
"dbserver1","mongodb.user": "debezium","mongodb.password":
"dbz","mongodb.task.id": "1","database.whitelist":
"inventory","database.history.pulsar.service.url": "pulsar://127.0.0.1:6650"}'
+ --archive
connectors/pulsar-io-debezium-mongodb-@pulsar:[email protected] \
+ --name debezium-mongodb-source \
+ --tenant public \
+ --namespace default \
+ --source-config '{"mongodb.hosts":
"rs0/mongodb:27017","mongodb.name": "dbserver1","mongodb.user":
"debezium","mongodb.password": "dbz","mongodb.task.id":
"1","database.whitelist": "inventory","database.history.pulsar.service.url":
"pulsar://127.0.0.1:6650"}'
```
:::note
@@ -519,7 +524,7 @@ This example shows how to change the data of a MongoDB
table using the Pulsar De
In the terminal window of subscribing topic, you can receive the following
messages.
- ```bash
+ ```text
----- got message -----
{"schema":{"type":"struct","fields":[{"type":"string","optional":false,"field":"id"}],"optional":false,"name":"dbserver1.inventory.products.Key"},"payload":{"id":"104"}},
value =
{"schema":{"type":"struct","fields":[{"type":"string","optional":true,"name":"io.debezium.data.Json","version":1,"field":"after"},{"type":"string","optional":true,"name":"io.debezium.data.Json","version":1,"field":"patch"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"version"},{"type"
[...]
```
diff --git a/docs/io-debug.md b/docs/io-debug.md
index 45a0907b34e..a99d1bbd6c0 100644
--- a/docs/io-debug.md
+++ b/docs/io-debug.md
@@ -64,12 +64,12 @@ For more information about the `localrun` command, see
[`localrun`](reference-co
```bash
./bin/pulsar-admin sinks localrun \
---archive connectors/pulsar-io-mongo-@pulsar:[email protected] \
---tenant public --namespace default \
---inputs test-mongo \
---name pulsar-mongo-sink \
---sink-config-file mongo-sink-config.yaml \
---parallelism 1
+ --archive connectors/pulsar-io-mongo-@pulsar:[email protected] \
+ --tenant public --namespace default \
+ --inputs test-mongo \
+ --name pulsar-mongo-sink \
+ --sink-config-file mongo-sink-config.yaml \
+ --parallelism 1
```
### Use connector log
@@ -215,20 +215,26 @@ Pulsar admin CLI helps you debug Pulsar connectors with
the following subcommand
```bash
./bin/pulsar-admin sinks create \
---archive pulsar-io-mongo-2.4.0.nar \
---tenant public \
---namespace default \
---inputs test-mongo \
---name pulsar-mongo-sink \
---sink-config-file mongo-sink-config.yaml \
---parallelism 1
+ --archive pulsar-io-mongo-2.4.0.nar \
+ --tenant public \
+ --namespace default \
+ --inputs test-mongo \
+ --name pulsar-mongo-sink \
+ --sink-config-file mongo-sink-config.yaml \
+ --parallelism 1
```
### `get`
+
Use the `get` command to get the basic information about the Mongo sink
connector, such as tenant, namespace, name, parallelism, and so on.
```bash
./bin/pulsar-admin sinks get --tenant public --namespace default --name
pulsar-mongo-sink
+```
+
+Output:
+
+```json
{
"tenant": "public",
"namespace": "default",
@@ -267,6 +273,11 @@ Use the `status` command to get the current status about
the Mongo sink connecto
--tenant public \
--namespace default \
--name pulsar-mongo-sink
+```
+
+Output:
+
+```json
{
"numInstances" : 1,
"numRunning" : 1,
@@ -297,10 +308,16 @@ If there are multiple connectors running on a worker,
`workerId` can locate the
:::
### `topics stats`
+
Use the `topics stats` command to get the stats for a topic and its connected
producer and consumer, such as whether the topic has received messages or not,
whether there is a backlog of messages or not, the available permits and other
key information. All rates are computed over a 1-minute window and are relative
to the last completed 1-minute period.
```bash
./bin/pulsar-admin topics stats test-mongo
+```
+
+Output:
+
+```json
{
"msgRateIn" : 0.0,
"msgThroughputIn" : 0.0,
diff --git a/docs/io-dynamodb-source.md b/docs/io-dynamodb-source.md
index 1205e4c9bed..1b17725d34b 100644
--- a/docs/io-dynamodb-source.md
+++ b/docs/io-dynamodb-source.md
@@ -4,6 +4,12 @@ title: AWS DynamoDB source connector
sidebar_label: "AWS DynamoDB source connector"
---
+:::note
+
+You can download all the Pulsar connectors on [download
page](https://pulsar.apache.org/download).
+
+::::
+
The DynamoDB source connector pulls data from DynamoDB table streams and
persists data into Pulsar.
This connector uses the [DynamoDB Streams Kinesis
Adapter](https://github.com/awslabs/dynamodb-streams-kinesis-adapter),
diff --git a/docs/io-elasticsearch-sink.md b/docs/io-elasticsearch-sink.md
index edb9adf7f14..62e8ae51df1 100644
--- a/docs/io-elasticsearch-sink.md
+++ b/docs/io-elasticsearch-sink.md
@@ -4,6 +4,12 @@ title: Elasticsearch sink connector
sidebar_label: "Elasticsearch sink connector"
---
+:::note
+
+You can download all the Pulsar connectors on [download
page](https://pulsar.apache.org/download).
+
+::::
+
The Elasticsearch sink connector pulls messages from Pulsar topics and
persists the messages to indexes.
## Requirements
diff --git a/docs/io-file-source.md b/docs/io-file-source.md
index 28615287aff..958be12aa39 100644
--- a/docs/io-file-source.md
+++ b/docs/io-file-source.md
@@ -4,6 +4,12 @@ title: File source connector
sidebar_label: "File source connector"
---
+:::note
+
+You can download all the Pulsar connectors on [download
page](https://pulsar.apache.org/download).
+
+::::
+
The File source connector pulls messages from files in directories and
persists the messages to Pulsar topics.
## Configuration
@@ -119,12 +125,12 @@ Here is an example of using the File source connecter.
```bash
docker exec -it pulsar-standalone /bin/bash
-
+
./bin/pulsar-admin sources localrun \
- --archive /pulsar/connectors/pulsar-io-file-{version}.nar \
- --name file-test \
- --destination-topic-name pulsar-file-test \
- --source-config-file /pulsar/file-connector.yaml
+ --archive /pulsar/connectors/pulsar-io-file-{version}.nar \
+ --name file-test \
+ --destination-topic-name pulsar-file-test \
+ --source-config-file /pulsar/file-connector.yaml
```
8. Start a consumer.
diff --git a/docs/io-flume-sink.md b/docs/io-flume-sink.md
index 03a49381880..e551ad57d06 100644
--- a/docs/io-flume-sink.md
+++ b/docs/io-flume-sink.md
@@ -4,6 +4,12 @@ title: Flume sink connector
sidebar_label: "Flume sink connector"
---
+:::note
+
+You can download all the Pulsar connectors on [download
page](https://pulsar.apache.org/download).
+
+::::
+
The Flume sink connector pulls messages from Pulsar topics to logs.
## Configuration
diff --git a/docs/io-flume-source.md b/docs/io-flume-source.md
index 9d362b7b92c..08919d43e40 100644
--- a/docs/io-flume-source.md
+++ b/docs/io-flume-source.md
@@ -4,6 +4,12 @@ title: Flume source connector
sidebar_label: "Flume source connector"
---
+:::note
+
+You can download all the Pulsar connectors on [download
page](https://pulsar.apache.org/download).
+
+::::
+
The Flume source connector pulls messages from logs to Pulsar topics.
## Configuration
diff --git a/docs/io-hbase-sink.md b/docs/io-hbase-sink.md
index 3345b859c92..f723ca976d4 100644
--- a/docs/io-hbase-sink.md
+++ b/docs/io-hbase-sink.md
@@ -4,6 +4,12 @@ title: HBase sink connector
sidebar_label: "HBase sink connector"
---
+:::note
+
+You can download all the Pulsar connectors on [download
page](https://pulsar.apache.org/download).
+
+::::
+
The HBase sink connector pulls the messages from Pulsar topics and persists
the messages to HBase tables
## Configuration
diff --git a/docs/io-hdfs2-sink.md b/docs/io-hdfs2-sink.md
index 05ab4ab8572..06ffd003c3a 100644
--- a/docs/io-hdfs2-sink.md
+++ b/docs/io-hdfs2-sink.md
@@ -4,6 +4,12 @@ title: HDFS2 sink connector
sidebar_label: "HDFS2 sink connector"
---
+:::note
+
+You can download all the Pulsar connectors on [download
page](https://pulsar.apache.org/download).
+
+::::
+
The HDFS2 sink connector pulls the messages from Pulsar topics and persists
the messages to HDFS files.
## Configuration
diff --git a/docs/io-hdfs3-sink.md b/docs/io-hdfs3-sink.md
index c487065b418..eff9bdfca42 100644
--- a/docs/io-hdfs3-sink.md
+++ b/docs/io-hdfs3-sink.md
@@ -4,6 +4,12 @@ title: HDFS3 sink connector
sidebar_label: "HDFS3 sink connector"
---
+:::note
+
+You can download all the Pulsar connectors on [download
page](https://pulsar.apache.org/download).
+
+::::
+
The HDFS3 sink connector pulls the messages from Pulsar topics and persists
the messages to HDFS files.
## Configuration
diff --git a/docs/io-http-sink.md b/docs/io-http-sink.md
index 7139a9a256d..4d5a72ea1c2 100644
--- a/docs/io-http-sink.md
+++ b/docs/io-http-sink.md
@@ -4,6 +4,12 @@ title: HTTTP sink connector
sidebar_label: "HTTP sink connector"
---
+:::note
+
+You can download all the Pulsar connectors on [download
page](https://pulsar.apache.org/download).
+
+::::
+
The HTTP sink connector pulls the records from Pulsar topics and makes a POST
request to a configurable HTTP URL (webhook).
The body of the HTTP request is the JSON representation of the record value.
The header `Content-Type: application/json` is added to the HTTP request.
diff --git a/docs/io-influxdb-sink.md b/docs/io-influxdb-sink.md
index 43c1d6cac23..89c93e4740d 100644
--- a/docs/io-influxdb-sink.md
+++ b/docs/io-influxdb-sink.md
@@ -4,6 +4,12 @@ title: InfluxDB sink connector
sidebar_label: "InfluxDB sink connector"
---
+:::note
+
+You can download all the Pulsar connectors on [download
page](https://pulsar.apache.org/download).
+
+::::
+
The InfluxDB sink connector pulls messages from Pulsar topics and persists the
messages to InfluxDB.
The InfluxDB sink provides different configurations for InfluxDBv1 and v2
respectively.
diff --git a/docs/io-jdbc-sink.md b/docs/io-jdbc-sink.md
index 63150517e28..6c61f1f6e16 100644
--- a/docs/io-jdbc-sink.md
+++ b/docs/io-jdbc-sink.md
@@ -4,6 +4,12 @@ title: JDBC sink connector
sidebar_label: "JDBC sink connector"
---
+:::note
+
+You can download all the Pulsar connectors on [download
page](https://pulsar.apache.org/download).
+
+::::
+
The JDBC sink connectors allow pulling messages from Pulsar topics and persist
the messages to ClickHouse, MariaDB, PostgreSQL, and SQLite.
> Currently, INSERT, DELETE and UPDATE operations are supported.
diff --git a/docs/io-kafka-sink.md b/docs/io-kafka-sink.md
index 3506ba47319..587cb2d2abd 100644
--- a/docs/io-kafka-sink.md
+++ b/docs/io-kafka-sink.md
@@ -4,6 +4,12 @@ title: Kafka sink connector
sidebar_label: "Kafka sink connector"
---
+:::note
+
+You can download all the Pulsar connectors on [download
page](https://pulsar.apache.org/download).
+
+::::
+
The Kafka sink connector pulls messages from Pulsar topics and persists the
messages to Kafka topics.
This guide explains how to configure and use the Kafka sink connector.
diff --git a/docs/io-kafka-source.md b/docs/io-kafka-source.md
index 44f5016df91..a4b47606b88 100644
--- a/docs/io-kafka-source.md
+++ b/docs/io-kafka-source.md
@@ -4,6 +4,12 @@ title: Kafka source connector
sidebar_label: "Kafka source connector"
---
+:::note
+
+You can download all the Pulsar connectors on [download
page](https://pulsar.apache.org/download).
+
+::::
+
The Kafka source connector pulls messages from Kafka topics and persists the
messages to Pulsar topics.
This guide explains how to configure and use the Kafka source connector.
@@ -85,12 +91,12 @@ Before using the Kafka source connector, you need to create
a configuration file
- YAML
```yaml
- configs:
- bootstrapServers: "pulsar-kafka:9092"
- groupId: "test-pulsar-io"
- topic: "my-topic"
- sessionTimeoutMs: "10000"
- autoCommitEnabled: false
+ configs:
+ bootstrapServers: "pulsar-kafka:9092"
+ groupId: "test-pulsar-io"
+ topic: "my-topic"
+ sessionTimeoutMs: "10000"
+ autoCommitEnabled: false
```
## Usage
@@ -156,15 +162,15 @@ This example describes how to use the Kafka source
connector to feed data from K
```bash
docker exec -it pulsar-kafka-standalone /bin/bash
-
+
./bin/pulsar-admin source localrun \
- --archive ./pulsar-io-kafka.nar \
- --tenant public \
- --namespace default \
- --name kafka \
- --destination-topic-name my-topic \
- --source-config-file ./conf/kafkaSourceConfig.yaml \
- --parallelism 1
+ --archive ./pulsar-io-kafka.nar \
+ --tenant public \
+ --namespace default \
+ --name kafka \
+ --destination-topic-name my-topic \
+ --source-config-file ./conf/kafkaSourceConfig.yaml \
+ --parallelism 1
```
7. Open a new terminal window and run the Kafka producer locally.
@@ -195,7 +201,7 @@ This example explains how to create a Kafka source
connector in an on-premises c
cp pulsar-io-kafka-{{connector:version}}.nar
$PULSAR_HOME/connectors/pulsar-io-kafka-{{connector:version}}.nar
```
-2. Reload all [built-in connectors](/io-connectors.md).
+2. Reload all [built-in connectors](io-connectors.md).
```bash
PULSAR_HOME/bin/pulsar-admin sources reload
diff --git a/docs/io-kinesis-sink.md b/docs/io-kinesis-sink.md
index fd486ad0c8f..11aae8a28b3 100644
--- a/docs/io-kinesis-sink.md
+++ b/docs/io-kinesis-sink.md
@@ -4,6 +4,12 @@ title: Kinesis sink connector
sidebar_label: "Kinesis sink connector"
---
+:::note
+
+You can download all the Pulsar connectors on [download
page](https://pulsar.apache.org/download).
+
+::::
+
The Kinesis sink connector pulls data from Pulsar and persists data into
Amazon Kinesis.
## Configuration
diff --git a/docs/io-kinesis-source.md b/docs/io-kinesis-source.md
index 469e7bcb574..e66fada7b12 100644
--- a/docs/io-kinesis-source.md
+++ b/docs/io-kinesis-source.md
@@ -4,6 +4,12 @@ title: Kinesis source connector
sidebar_label: "Kinesis source connector"
---
+:::note
+
+You can download all the Pulsar connectors on [download
page](https://pulsar.apache.org/download).
+
+::::
+
The Kinesis source connector pulls data from Amazon Kinesis and persists data
into Pulsar.
This connector uses the [Kinesis Consumer
Library](https://github.com/awslabs/amazon-kinesis-client) (KCL) to do the
actual consumption of messages. The KCL uses DynamoDB to track the state of
consumers.
diff --git a/docs/io-mongo-sink.md b/docs/io-mongo-sink.md
index e55092c3ab7..954b6109683 100644
--- a/docs/io-mongo-sink.md
+++ b/docs/io-mongo-sink.md
@@ -4,6 +4,12 @@ title: MongoDB sink connector
sidebar_label: "MongoDB sink connector"
---
+:::note
+
+You can download all the Pulsar connectors on [download
page](https://pulsar.apache.org/download).
+
+::::
+
The MongoDB sink connector pulls messages from Pulsar topics and persists the
messages to collections.
## Configuration
diff --git a/docs/io-mongo-source.md b/docs/io-mongo-source.md
index a525cb79b32..ae74d6be53f 100644
--- a/docs/io-mongo-source.md
+++ b/docs/io-mongo-source.md
@@ -4,6 +4,12 @@ title: MongoDB source connector
sidebar_label: "MongoDB source connector"
---
+:::note
+
+You can download all the Pulsar connectors on [download
page](https://pulsar.apache.org/download).
+
+::::
+
The MongoDB source connector pulls documents from MongoDB and persists the
messages to Pulsar topics.
This guide explains how to configure and use the MongoDB source connector.
diff --git a/docs/io-netty-source.md b/docs/io-netty-source.md
index 5a164539afa..95542636935 100644
--- a/docs/io-netty-source.md
+++ b/docs/io-netty-source.md
@@ -4,6 +4,12 @@ title: Netty source connector
sidebar_label: "Netty source connector"
---
+:::note
+
+You can download all the Pulsar connectors on [download
page](https://pulsar.apache.org/download).
+
+::::
+
The Netty source connector opens a port that accepts incoming data via the
configured network protocol and publishes it to user-defined Pulsar topics.
This connector can be used in a containerized (for example, k8s) deployment.
Otherwise, if the connector is running in process or thread mode, the instance
may be conflicting on listening to ports.
diff --git a/docs/io-nsq-source.md b/docs/io-nsq-source.md
index 88c88c5e636..0e67ac1404c 100644
--- a/docs/io-nsq-source.md
+++ b/docs/io-nsq-source.md
@@ -4,6 +4,12 @@ title: NSQ source connector
sidebar_label: "NSQ source connector"
---
+:::note
+
+You can download all the Pulsar connectors on [download
page](https://pulsar.apache.org/download).
+
+::::
+
The NSQ source connector receives messages from NSQ topics
and writes messages to Pulsar topics.
diff --git a/docs/io-rabbitmq-sink.md b/docs/io-rabbitmq-sink.md
index a56b8164244..265992737e7 100644
--- a/docs/io-rabbitmq-sink.md
+++ b/docs/io-rabbitmq-sink.md
@@ -4,6 +4,12 @@ title: RabbitMQ sink connector
sidebar_label: "RabbitMQ sink connector"
---
+:::note
+
+You can download all the Pulsar connectors on [download
page](https://pulsar.apache.org/download).
+
+::::
+
The RabbitMQ sink connector pulls messages from Pulsar topics and persists the
messages to RabbitMQ queues.
diff --git a/docs/io-rabbitmq-source.md b/docs/io-rabbitmq-source.md
index fce240bc15d..012a6aba2b8 100644
--- a/docs/io-rabbitmq-source.md
+++ b/docs/io-rabbitmq-source.md
@@ -4,6 +4,12 @@ title: RabbitMQ source connector
sidebar_label: "RabbitMQ source connector"
---
+:::note
+
+You can download all the Pulsar connectors on [download
page](https://pulsar.apache.org/download).
+
+::::
+
The RabbitMQ source connector receives messages from RabbitMQ clusters and
writes messages to Pulsar topics.
## Configuration
diff --git a/docs/io-redis-sink.md b/docs/io-redis-sink.md
index ea5f51a8d23..9b8ed060b94 100644
--- a/docs/io-redis-sink.md
+++ b/docs/io-redis-sink.md
@@ -4,9 +4,13 @@ title: Redis sink connector
sidebar_label: "Redis sink connector"
---
-The Redis sink connector pulls messages from Pulsar topics and persists the
messages to a Redis database.
+:::note
+
+You can download all the Pulsar connectors on [download
page](https://pulsar.apache.org/download).
+::::
+The Redis sink connector pulls messages from Pulsar topics and persists the
messages to a Redis database.
## Configuration
@@ -90,27 +94,27 @@ This example shows how to write records to a Redis database
using the Pulsar Red
* Use the **JSON** configuration file as shown previously.
- ```bash
- bin/pulsar-admin sinks localrun \
+ ```bash
+ bin/pulsar-admin sinks localrun \
--archive connectors/pulsar-io-redis-@pulsar:[email protected] \
--tenant public \
--namespace default \
--name my-redis-sink \
--sink-config '{"redisHosts": "localhost:6379","redisPassword":
"mypassword","redisDatabase": "0","clientMode":
"Standalone","operationTimeout": "3000","batchSize": "1"}' \
--inputs my-redis-topic
- ```
+ ```
* Use the **YAML** configuration file as shown previously.
- ```bash
- bin/pulsar-admin sinks localrun \
- --archive connectors/pulsar-io-redis-@pulsar:[email protected] \
- --tenant public \
- --namespace default \
- --name my-redis-sink \
- --sink-config-file redis-sink-config.yaml \
- --inputs my-redis-topic
- ```
+ ```bash
+ bin/pulsar-admin sinks localrun \
+ --archive connectors/pulsar-io-redis-@pulsar:[email protected] \
+ --tenant public \
+ --namespace default \
+ --name my-redis-sink \
+ --sink-config-file redis-sink-config.yaml \
+ --inputs my-redis-topic
+ ```
4. Publish records to the topic.
diff --git a/docs/io-solr-sink.md b/docs/io-solr-sink.md
index f4ecd2b34b2..9d288cd345e 100644
--- a/docs/io-solr-sink.md
+++ b/docs/io-solr-sink.md
@@ -4,9 +4,13 @@ title: Solr sink connector
sidebar_label: "Solr sink connector"
---
-The Solr sink connector pulls messages from Pulsar topics and persists the
messages to Solr collections.
+:::note
+
+You can download all the Pulsar connectors on [download
page](https://pulsar.apache.org/download).
+::::
+The Solr sink connector pulls messages from Pulsar topics and persists the
messages to Solr collections.
## Configuration
@@ -49,13 +53,12 @@ Before using the Solr sink connector, you need to create a
configuration file th
* YAML
```yaml
- {
- solrUrl: "localhost:2181,localhost:2182/chroot"
- solrMode: "SolrCloud"
- solrCollection: "techproducts"
- solrCommitWithinMs: 100
- username: "fakeuser"
- password: "fake@123"
- }
+ configs:
+ solrUrl: "localhost:2181,localhost:2182/chroot"
+ solrMode: "SolrCloud"
+ solrCollection: "techproducts"
+ solrCommitWithinMs: 100
+ username: "fakeuser"
+ password: "fake@123"
```
diff --git a/docs/io-twitter-source.md b/docs/io-twitter-source.md
index 6cb0a09e2b7..3cf2539201c 100644
--- a/docs/io-twitter-source.md
+++ b/docs/io-twitter-source.md
@@ -4,6 +4,12 @@ title: Twitter Firehose source connector
sidebar_label: "Twitter Firehose source connector"
---
+:::note
+
+You can download all the Pulsar connectors on [download
page](https://pulsar.apache.org/download).
+
+::::
+
The Twitter Firehose source connector receives tweets from Twitter Firehose
and
writes the tweets to Pulsar topics.
