Re: Writing the contents of spark dataframe to Kafka with Spark 2.2

fka). > > BR, > G > > > On Tue, Mar 19, 2019 at 4:35 PM anna stax wrote: > >> Hi Gabor, >> >> Thank you for the response. >> >> I do have those dependencies added. >> >> >> org.apache.spark >> spark-core_2.11 >>

Re: Writing the contents of spark dataframe to Kafka with Spark 2.2

d with spark-shell as well... > > BR, > G > > > On Mon, Mar 18, 2019 at 10:07 PM anna stax wrote: > >> Hi all, >> I am unable to write the contents of spark dataframe to Kafka. >> I am using Spark 2.2 >> >> This is my code >> >> val df = Seq(

Writing the contents of spark dataframe to Kafka with Spark 2.2

Hi all, I am unable to write the contents of spark dataframe to Kafka. I am using Spark 2.2 This is my code val df = Seq(("1","One"),("2","two")).toDF("key","value") df.printSchema() df.show(false) df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)") .write .format("kafka")

Re: Unable to see the table created using saveAsTable From Beeline. Please help!

Is some configuration missing ? Appreciate any help On Fri, Jul 6, 2018 at 4:10 PM, anna stax wrote: > I am running spark 2.1.0 on AWS EMR > > In my Zeppelin Note I am creating a table > > df.write > .format("parquet") > .saveAsTable("default.

Unable to see the table created using saveAsTable From Beeline. Please help!

I am running spark 2.1.0 on AWS EMR In my Zeppelin Note I am creating a table df.write .format("parquet") .saveAsTable("default.1test") and I see the table when I spark.catalog.listTables().show() +++---+-+---+ |

Re: How do I save the dataframe data as a pdf file?

Thanks Anthony for the response. Yes, the data in the dataframe represents a report and I want to create pdf files. I am using scala so hoping to find a easier solution in scala, if not I will try out your suggestion . On Tue, Dec 12, 2017 at 11:29 AM, Anthony Thomas

Re: Spark streaming for CEP

; ingestion of data then Spark streaming may not be granular enough for CEP. >>>>> You may consider other products. >>>>> >>>>> Worth looking at this old thread on mine "Spark support for Complex >>>>> Event Processing (CEP) &g

Spark streaming for CEP

Hello all, Has anyone used spark streaming for CEP (Complex Event processing). Any CEP libraries that works well with spark. I have a use case for CEP and trying to see if spark streaming is a good fit. Currently we have a data pipeline using Kafka, Spark streaming and Cassandra for data

Re: Spark streaming, Storage tab questions

On Sun, Jul 9, 2017 at 4:33 PM, anna stax <annasta...@gmail.com> wrote: > Does each row represent the state of my app at different time? > > When the fraction cached is 90% and the size on Disk is 0, does that mean > 10% of the data is lost. Its neither in memory now disk? >

Logging in lSpark streaming application

Do I need to include the log4j dependencies in my pom.xml of the spark streaming application or it is already included in spark libraries? I am running Spark in standalone mode on AWS EC2. Thanks

Spark standalone , client mode. How do I monitor?

Hi all, I have a spark standalone cluster. I am running a spark streaming application on it and the deploy mode is client. I am looking for the best way to monitor the cluster and application so that I will know when the application/cluster is down. I cannot move to cluster deploy mode now. I

Re: help/suggestions to setup spark cluster

information of what you are trying to do but if you > >> are just trying to set things up from scratch then I think you can just > use > >> EMR which will create a cluster for you and attach a zeppelin instance > as > >> well > >> > >> > >>

Re: help/suggestions to setup spark cluster

ittle management but > you will pay a premium for that abstraction > > > Regards > Sam > On Wed, 26 Apr 2017 at 22:02, anna stax <annasta...@gmail.com> wrote: > >> I need to setup a spark cluster for Spark streaming and scheduled batch >> jobs and ad

help/suggestions to setup spark cluster

I need to setup a spark cluster for Spark streaming and scheduled batch jobs and adhoc queries. Please give me some suggestions. Can this be done in standalone mode. Right now we have a spark cluster in standalone mode on AWS EC2 running spark streaming application. Can we run spark batch jobs