Hi Kant, Kafka is the message broker that using as Producers and Consumers and Spark Streaming is used as the real time processing ,Kafka and Spark Streaming work together not competitors. Spark Streaming is reading data from Kafka and process into micro batching for streaming data, In easy terms collects data for some time, build RDD and then process these micro batches.
Please read doc : https://spark.apache.org/docs/latest/streaming-programming-guide.html Spark Streaming is an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams. Data can be ingested from many sources like *Kafka, Flume, Kinesis, or TCP sockets*, and can be processed using complex algorithms expressed with high-level functions like map, reduce, join and window. Finally, processed data can be pushed out to filesystems, databases, and live dashboards. In fact, you can apply Spark’s machine learning <https://spark.apache.org/docs/latest/ml-guide.html> and graph processing <https://spark.apache.org/docs/latest/graphx-programming-guide.html> algorithms on data streams. Regards, Vaquar khan On Sun, Jun 11, 2017 at 3:12 AM, kant kodali <kanth...@gmail.com> wrote: > Hi All, > > I am trying hard to figure out what is the real difference between Kafka > Streaming vs Spark Streaming other than saying one can be used as part of > Micro services (since Kafka streaming is just a library) and the other is a > Standalone framework by itself. > > If I can accomplish same job one way or other this is a sort of a puzzling > question for me so it would be great to know what Spark streaming can do > that Kafka Streaming cannot do efficiently or whatever ? > > Thanks! > > -- Regards, Vaquar Khan +1 -224-436-0783 Greater Chicago
