junaiddshaukat commented on code in PR #39627: URL: https://github.com/apache/beam/pull/39627#discussion_r3736360573
########## website/www/site/content/en/documentation/runners/kafkastreams.md: ########## @@ -0,0 +1,146 @@ +--- +type: runners +title: "Kafka Streams Runner" +--- +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Kafka Streams Runner + +The Kafka Streams Runner executes Beam pipelines on [Kafka +Streams](https://kafka.apache.org/documentation/streams/), by translating a pipeline into a Kafka +Streams topology. + +What distinguishes it from the other runners is that Kafka Streams is a library rather than a +cluster. There is no job manager and no resource manager to operate: an application is an ordinary +JVM process that reads from and writes to Kafka, and scaling it means starting more copies of that +process. Fault tolerance, state, and exactly-once processing come from Kafka itself — from consumer +groups, changelog topics, and transactions. + +That makes it worth considering if you already run Kafka and want Beam's programming model without +introducing a second distributed system to operate. + +## The runner is experimental + +**The Kafka Streams Runner is experimental.** It executes a meaningful subset of the Beam model +correctly, and the parts it does support are covered by Beam's own `@ValidatesRunner` suite, but +several capabilities that are core to the model are not implemented yet. Read [what is not +supported](#what-is-not-supported-yet) before choosing it for anything real. + +It is also aimed squarely at streaming. A pipeline over bounded data will run, but there are more +efficient choices for batch work; this runner exists for pipelines that do not end. + +## Running a pipeline + +The runner is portable — it executes user code over the Fn API, in an SDK harness — so a pipeline is +submitted to a job server rather than run directly. + +Start the job server, which listens on `localhost:8099` by default: + +``` +./gradlew :runners:kafka-streams:runJobServer +``` + +Then submit a pipeline against it, pointing the runner at your Kafka cluster: + +``` +--runner=PortableRunner \ +--jobEndpoint=localhost:8099 \ +--bootstrapServers=localhost:9092 \ +--applicationId=my-beam-pipeline Review Comment: Added the Python section here, since the page lives in this PR. It documents --runner=KafkaStreamsRunner from Python — the wrapper builds the job server jar, starts it, and stops it with the pipeline — plus LOOPBACK meaning no Docker is needed locally, and --kafka_streams_job_server_jar for a prebuilt jar. The wrapper itself is in #39627 Writing it turned up a mistake in what I already had: the shared-job-server example mixed Python's --job_endpoint with Java's --bootstrapServers, which wouldn't have worked as written. That section now shows each SDK separately, and the options table notes the Python spelling is the same in snake case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
