Repository: samza Updated Branches: refs/heads/master 6b3189436 -> 4baaddbbb
Adding connectors overview doc Author: Aditya Toomula <atoom...@linkedin.com> Reviewers: Jagadish<jagad...@apache.org> Closes #689 from atoomula/connector Project: http://git-wip-us.apache.org/repos/asf/samza/repo Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/4baaddbb Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/4baaddbb Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/4baaddbb Branch: refs/heads/master Commit: 4baaddbbbb665fefac821eb0aecd1608d5a6bdf4 Parents: 6b31894 Author: Aditya Toomula <atoom...@linkedin.com> Authored: Fri Oct 5 11:02:45 2018 -0700 Committer: Jagadish <jvenkatra...@linkedin.com> Committed: Fri Oct 5 11:02:45 2018 -0700 ---------------------------------------------------------------------- .../documentation/versioned/api/table-api.md | 2 +- .../versioned/connectors/overview.md | 32 ++++++++++++++++++-- 2 files changed, 31 insertions(+), 3 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/samza/blob/4baaddbb/docs/learn/documentation/versioned/api/table-api.md ---------------------------------------------------------------------- diff --git a/docs/learn/documentation/versioned/api/table-api.md b/docs/learn/documentation/versioned/api/table-api.md index d95dfac..f5efa88 100644 --- a/docs/learn/documentation/versioned/api/table-api.md +++ b/docs/learn/documentation/versioned/api/table-api.md @@ -191,7 +191,7 @@ The code snippet below illustrates the usage of table in Samza low level API. 5 DelegatingSystemDescriptor ksd = new DelegatingSystemDescriptor("mySystem"); 6 7 TableDescriptor<Integer, Profile> tableDesc = new InMemoryTableDescriptor( - 8 "T1", KVSerde.of(new IntegerSerde(), new ProfileJsonSerde())); + 8 "t1", KVSerde.of(new IntegerSerde(), new ProfileJsonSerde())); 9 GenericInputDescriptor<Profile> profileISD = ksd.getInputDescriptor("Profile", new NoOpSerde<>()); 10 11 appDesc.addTable(tableDesc); http://git-wip-us.apache.org/repos/asf/samza/blob/4baaddbb/docs/learn/documentation/versioned/connectors/overview.md ---------------------------------------------------------------------- diff --git a/docs/learn/documentation/versioned/connectors/overview.md b/docs/learn/documentation/versioned/connectors/overview.md index 579c494..5b6ba39 100644 --- a/docs/learn/documentation/versioned/connectors/overview.md +++ b/docs/learn/documentation/versioned/connectors/overview.md @@ -19,6 +19,34 @@ title: Connectors overview limitations under the License. --> -# Section 1 -# Section 2 +Stream processing applications often read data from external sources like Kafka or HDFS. Likewise, they require processed +results to be written to external system or data stores. As of the 1.0 release, Samza integrates with the following systems +out-of-the-box: + +- [Apache Kafka](kafka) (consumer/producer) +- [Microsoft Azure Eventhubs](eventhubs) (consumer/producer) +- [Amazon AWS Kinesis Streams](kinesis) (consumer) +- [Hadoop Filesystem](hdfs) (consumer/producer) +- [Elasticsearch](https://github.com/apache/samza/blob/master/samza-elasticsearch/src/main/java/org/apache/samza/system/elasticsearch/ElasticsearchSystemProducer.java) (producer) + +Instructions on how to use these connectors can be found in the corresponding subsections. Please note that the +connector API is different from [Samza Table API](../api/table-api), where the data could be read from and written to +data stores. + +Samza is pluggable and designed to support a variety of producers and consumers. You can provide your own producer or +consumer by implementing the SystemFactory interface. + +To associate a system with a Samza Connector, the user needs to set the following config: + +{% highlight jproperties %} +systems.<system-name>.samza.factory=org.apache.samza.system.kafka.KafkaSystemFactory +{% endhighlight %} + +Any system specific configs, could be defined as below: + +{% highlight jproperties %} +systems.<system-name>.param1=value1 +systems.<system-name>.consumer.param2=value2 +systems.<system-name>.producer.param3=value3 +{% endhighlight %}