Is it OK to use ProtoBuf for sending messages to Kafka? I do not see
anyone using it .
Please direct me to some code samples of how to use it in Spark Structured
streaming.
Thanks again..
On Sat, Nov 12, 2016 at 11:44 PM, shyla deshpande
wrote:
> Thanks everyone. Very good discussion.
>
> Th
Thanks everyone. Very good discussion.
Thanks Jacek, for the code snippet. I downloaded your Mastering Apache
Spark pdf . I love it.
I have one more question,
On Sat, Nov 12, 2016 at 2:21 PM, Sean McKibben wrote:
> I think one of the advantages of using akka-streams within Spark is the
> fact
I think one of the advantages of using akka-streams within Spark is the fact
that it is a general purpose stream processing toolset with backpressure, not
necessarily specific to kafka. If things work out with the approach, Spark
could be a great benefit to use as a coordination framework for di
Hi Luciano,
Mind sharing why to have a structured streaming source/sink for Akka
if Kafka's available and Akka Streams has a Kafka module? #curious
Pozdrawiam,
Jacek Laskowski
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 https://bit.ly/mastering-apache-spark
Follow me at ht
If you are interested in Akka streaming, it is being maintained in Apache
Bahir. For Akka there isn't a structured streaming version yet, but we
would be interested in collaborating in the structured streaming version
for sure.
On Thu, Nov 10, 2016 at 8:46 AM shyla deshpande
wrote:
> I am using
Hi,
Just to add to Cody's answer...the following snippet works fine on master:
spark.readStream
.format("kafka")
.option("subscribe", "topic")
.option("kafka.bootstrap.servers", "localhost:9092")
.load
.writeStream
.format("console")
.start
Don't forget to add spark-sql-kafka-0-10
The basic structured streaming source for Kafka is already committed to
master, build it and try it out.
If you're already using Kafka I don't really see much point in trying to
put Akka in between it and Spark.
On Nov 10, 2016 02:25, "vincent gromakowski"
wrote:
I have already integrated commo
I have already integrated common actors. I am also interested, specially to
see how we can achieve end to end back pressure.
2016-11-10 8:46 GMT+01:00 shyla deshpande :
> I am using Spark 2.0.1. I wanted to build a data pipeline using Kafka,
> Spark Streaming and Cassandra using Structured Stream
I am using Spark 2.0.1. I wanted to build a data pipeline using Kafka,
Spark Streaming and Cassandra using Structured Streaming. But the kafka
source support for Structured Streaming is not yet available. So now I am
trying to use Akka Stream as the source to Spark Streaming.
Want to make sure I a
