FOSS == Free Open Source Software From: andrew davidson <a...@santacruzanalytics.com> Date: Wednesday, March 30, 2022 at 3:16 PM To: users@kafka.apache.org <users@kafka.apache.org> Subject: [EXTERNAL] Re: Newbie looking for a connector I can configure on my mac Thanks Liam.
What is 'FOSS Kafka'? google did not find any useful definitions A tutorial would be great! I would be happy to provide feed back https://urldefense.com/v3/__https://docs.confluent.io/platform/current/connect/filestream_connector.html__;!!K_cMf-SQz-o!KwauyHePgwgsmQcFKSAVUaFQKtNg3E4x_56EGI-NS2nziM4tnO2dhNzU6uIPjCUe9wex7w$<https://urldefense.com/v3/__https:/docs.confluent.io/platform/current/connect/filestream_connector.html__;!!K_cMf-SQz-o!KwauyHePgwgsmQcFKSAVUaFQKtNg3E4x_56EGI-NS2nziM4tnO2dhNzU6uIPjCUe9wex7w$> looks really helpful. Your list of connectors is very helpful. I wonder if your email would be a good thing to add to the kafka documentation quick start guide or FAQ? Kind regards Andy On 3/29/22, 5:27 PM, "Liam Clarke-Hutchinson" <lclar...@redhat.com> wrote: Hi Andrew, So if you've downloaded Apache Kafka, you can run a standalone connect instance using the bin/connect-standalone.sh script mentioned. And while a lot of the connector documentation is on the Confluent website, you can still use them with FOSS Kafka so long as you're in line with the Confluent Community Licence (basically, IIRC, you can use them for free, but not to run a SAAS or similar that competes with Confluent, but IANAL). I agree that there's not much useful documentation for your use case. I will look into writing a tutorial for your use case, would you be happy to give me feedback on it as I go? The most important configuration initially is the plugin.path, where your standalone KC process will look for those JARs. You can see an example properties file for standalone Connect under the config/ dir in the Kafka you downloaded. Note that it has the plugin path commented out initially. So, Kafka ships with a connector that exposes a file source and file sink, which is good for testing out KC and getting used to it. You can either build it from source, or download it from here: https://urldefense.com/v3/__https://mvnrepository.com/artifact/org.apache.kafka/connect-file__;!!K_cMf-SQz-o!KwauyHePgwgsmQcFKSAVUaFQKtNg3E4x_56EGI-NS2nziM4tnO2dhNzU6uIPjCW7qkQA0Q$<https://urldefense.com/v3/__https:/mvnrepository.com/artifact/org.apache.kafka/connect-file__;!!K_cMf-SQz-o!KwauyHePgwgsmQcFKSAVUaFQKtNg3E4x_56EGI-NS2nziM4tnO2dhNzU6uIPjCW7qkQA0Q$> - choose the version that matches the version of Kafka you've downloaded, and then you can download the JAR under files up the top. This documentation from Confluent is useful: https://urldefense.com/v3/__https://docs.confluent.io/platform/current/connect/filestream_connector.html__;!!K_cMf-SQz-o!KwauyHePgwgsmQcFKSAVUaFQKtNg3E4x_56EGI-NS2nziM4tnO2dhNzU6uIPjCUe9wex7w$<https://urldefense.com/v3/__https:/docs.confluent.io/platform/current/connect/filestream_connector.html__;!!K_cMf-SQz-o!KwauyHePgwgsmQcFKSAVUaFQKtNg3E4x_56EGI-NS2nziM4tnO2dhNzU6uIPjCUe9wex7w$> Note that if you don't provide a file property, (this isn't documented either(!)) it will use standard input for the file source, and standard output for the file sink. You can see example configs for this connector reading from a file or console under that same config/ directory, and ditto for writing. These connectors might also be useful for playing with KC, and are all free and downloadable: https://urldefense.com/v3/__https://www.confluent.io/hub/confluentinc/kafka-connect-datagen__;!!K_cMf-SQz-o!KwauyHePgwgsmQcFKSAVUaFQKtNg3E4x_56EGI-NS2nziM4tnO2dhNzU6uIPjCVucIx6Ng$<https://urldefense.com/v3/__https:/www.confluent.io/hub/confluentinc/kafka-connect-datagen__;!!K_cMf-SQz-o!KwauyHePgwgsmQcFKSAVUaFQKtNg3E4x_56EGI-NS2nziM4tnO2dhNzU6uIPjCVucIx6Ng$> <- generates a stream of test data https://urldefense.com/v3/__https://www.confluent.io/hub/jcustenborder/kafka-connect-twitter__;!!K_cMf-SQz-o!KwauyHePgwgsmQcFKSAVUaFQKtNg3E4x_56EGI-NS2nziM4tnO2dhNzU6uIPjCXfwK4JoQ$<https://urldefense.com/v3/__https:/www.confluent.io/hub/jcustenborder/kafka-connect-twitter__;!!K_cMf-SQz-o!KwauyHePgwgsmQcFKSAVUaFQKtNg3E4x_56EGI-NS2nziM4tnO2dhNzU6uIPjCXfwK4JoQ$> <- disregard, I saw you mentioned not having Twitter https://urldefense.com/v3/__https://www.confluent.io/hub/C0urante/kafka-connect-reddit__;!!K_cMf-SQz-o!KwauyHePgwgsmQcFKSAVUaFQKtNg3E4x_56EGI-NS2nziM4tnO2dhNzU6uIPjCUgRY66HQ$<https://urldefense.com/v3/__https:/www.confluent.io/hub/C0urante/kafka-connect-reddit__;!!K_cMf-SQz-o!KwauyHePgwgsmQcFKSAVUaFQKtNg3E4x_56EGI-NS2nziM4tnO2dhNzU6uIPjCUgRY66HQ$> <- I haven't used this, but could be interesting? I hope this helps you get started, and please let me know if I can help with anything else :) Cheers, Liam Clarke On Wed, 30 Mar 2022 at 11:54, andrew davidson <a...@santacruzanalytics.com> wrote: > I found the quick start https://urldefense.com/v3/__https://kafka.apache.org/quickstart__;!!K_cMf-SQz-o!KwauyHePgwgsmQcFKSAVUaFQKtNg3E4x_56EGI-NS2nziM4tnO2dhNzU6uIPjCVqUYeB3g$<https://urldefense.com/v3/__https:/kafka.apache.org/quickstart__;!!K_cMf-SQz-o!KwauyHePgwgsmQcFKSAVUaFQKtNg3E4x_56EGI-NS2nziM4tnO2dhNzU6uIPjCVqUYeB3g$> example very > helpful. It made it really easy to understand how download, start up, > create topic, push some data through the Kafka. I did not find > https://urldefense.com/v3/__https://kafka.apache.org/quickstart*quickstart_kafkaconnect__;Iw!!K_cMf-SQz-o!KwauyHePgwgsmQcFKSAVUaFQKtNg3E4x_56EGI-NS2nziM4tnO2dhNzU6uIPjCXxevE4wA$<https://urldefense.com/v3/__https:/kafka.apache.org/quickstart*quickstart_kafkaconnect__;Iw!!K_cMf-SQz-o!KwauyHePgwgsmQcFKSAVUaFQKtNg3E4x_56EGI-NS2nziM4tnO2dhNzU6uIPjCXxevE4wA$> useful. > > I am looking for something very simple to learning how to configure and > use connectors using Apache Kafka distribution, not Confluent. I can run on > my mac or Linux server. Being a newbie I want to keep things super simple. > I do not want to have to debug firewalls, ACL, … > > I do not have a data base, access to twitter, … > > I thought maybe something some sort source/sink using the local file > system? > > Any suggestions? > > Kind regards > > Andy > > p.s. I have read a lot of documentation most of it is very high level. Can > anyone recommend a “hand on” tutorial? > > > This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.
