Hi,
I have created the Kafka messaging architecture as a microservice that
feeds both Spark streaming and Flink. Spark streaming uses micro-batches
meaning "collect and process data" and flink as an event driven
architecture (a stateful application that reacts to incoming events by
triggering computations etc.
According to Wikipedia, A Microservice is a technique that structures an
application as a collection of loosely coupled services. In a microservices
architecture, services are fine-grained and the protocols are lightweight.
Ok for streaming data among other things I have to create and configure
topic (or topics), design a robust zookeeper ensemble and create Kafka
brokers with scalability and resiliency. Then I can offer the streaming as
a microservice to subscribers among them Spark and Flink. I can upgrade
this microservice component in isolation without impacting either Spark or
Flink.
The problem I face here is the dependency on Flink etc on the jar files
specific for the version of Kafka deployed. For example kafka_2.12-1.1.0 is
built on Scala 2.12 and Kafka version 1.1.0. To make this work in Flink 1.5
application, I need to use the correct dependency in sbt build. For
example:
libraryDependencies += "org.apache.flink" %% "flink-connector-kafka-0.11" %
"1.5.0"
libraryDependencies += "org.apache.flink" %% "flink-connector-kafka-base" %
"1.5.0"
libraryDependencies += "org.apache.flink" %% "flink-scala" % "1.5.0"
libraryDependencies += "org.apache.kafka" % "kafka-clients" % "0.11.0.0"
libraryDependencies += "org.apache.flink" %% "flink-streaming-scala" %
"1.5.0"
libraryDependencies += "org.apache.kafka" %% "kafka" % "0.11.0.0"
and the Scala code needs to change:
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer011
…
val stream = env
.addSource(new FlinkKafkaConsumer011[String]("md", new
SimpleStringSchema(), properties))
So in summary some changes need to be made to Flink to be able to interact
with the new version of Kafka. And more importantly if one can use an
abstract notion of microservice here?
Dr Mich Talebzadeh
LinkedIn *
https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
