help/suggestions to setup spark cluster
I need to setup a spark cluster for Spark streaming and scheduled batch jobs and adhoc queries. Please give me some suggestions. Can this be done in standalone mode. Right now we have a spark cluster in standalone mode on AWS EC2 running spark streaming application. Can we run spark batch jobs and zeppelin on the same. Do we need a better resource manager like Mesos? Are there any companies or individuals that can help in setting this up? Thank you. -Anna
Re: help/suggestions to setup spark cluster
Hi Sam, Thank you for the reply. What do you mean by I doubt people run spark in a. Single EC2 instance, certainly not in production I don't think What is wrong in having a data pipeline on EC2 that reads data from kafka, processes using spark and outputs to cassandra? Please explain. Thanks -Anna On Wed, Apr 26, 2017 at 2:22 PM, Sam Elamin wrote: > Hi Anna > > There are a variety of options for launching spark clusters. I doubt > people run spark in a. Single EC2 instance, certainly not in production I > don't think > > I don't have enough information of what you are trying to do but if you > are just trying to set things up from scratch then I think you can just use > EMR which will create a cluster for you and attach a zeppelin instance as > well > > > You can also use databricks for ease of use and very little management but > you will pay a premium for that abstraction > > > Regards > Sam > On Wed, 26 Apr 2017 at 22:02, anna stax wrote: > >> I need to setup a spark cluster for Spark streaming and scheduled batch >> jobs and adhoc queries. >> Please give me some suggestions. Can this be done in standalone mode. >> >> Right now we have a spark cluster in standalone mode on AWS EC2 running >> spark streaming application. Can we run spark batch jobs and zeppelin on >> the same. Do we need a better resource manager like Mesos? >> >> Are there any companies or individuals that can help in setting this up? >> >> Thank you. >> >> -Anna >> >
Re: help/suggestions to setup spark cluster
Thanks Cody, As I already mentioned I am running spark streaming on EC2 cluster in standalone mode. Now in addition to streaming, I want to be able to run spark batch job hourly and adhoc queries using Zeppelin. Can you please confirm that a standalone cluster is OK for this. Please provide me some links to help me get started. Thanks -Anna On Wed, Apr 26, 2017 at 7:46 PM, Cody Koeninger wrote: > The standalone cluster manager is fine for production. Don't use Yarn > or Mesos unless you already have another need for it. > > On Wed, Apr 26, 2017 at 4:53 PM, anna stax wrote: > > Hi Sam, > > > > Thank you for the reply. > > > > What do you mean by > > I doubt people run spark in a. Single EC2 instance, certainly not in > > production I don't think > > > > What is wrong in having a data pipeline on EC2 that reads data from > kafka, > > processes using spark and outputs to cassandra? Please explain. > > > > Thanks > > -Anna > > > > On Wed, Apr 26, 2017 at 2:22 PM, Sam Elamin > wrote: > >> > >> Hi Anna > >> > >> There are a variety of options for launching spark clusters. I doubt > >> people run spark in a. Single EC2 instance, certainly not in production > I > >> don't think > >> > >> I don't have enough information of what you are trying to do but if you > >> are just trying to set things up from scratch then I think you can just > use > >> EMR which will create a cluster for you and attach a zeppelin instance > as > >> well > >> > >> > >> You can also use databricks for ease of use and very little management > but > >> you will pay a premium for that abstraction > >> > >> > >> Regards > >> Sam > >> On Wed, 26 Apr 2017 at 22:02, anna stax wrote: > >>> > >>> I need to setup a spark cluster for Spark streaming and scheduled batch > >>> jobs and adhoc queries. > >>> Please give me some suggestions. Can this be done in standalone mode. > >>> > >>> Right now we have a spark cluster in standalone mode on AWS EC2 running > >>> spark streaming application. Can we run spark batch jobs and zeppelin > on the > >>> same. Do we need a better resource manager like Mesos? > >>> > >>> Are there any companies or individuals that can help in setting this > up? > >>> > >>> Thank you. > >>> > >>> -Anna > > > > >
Spark standalone , client mode. How do I monitor?
Hi all, I have a spark standalone cluster. I am running a spark streaming application on it and the deploy mode is client. I am looking for the best way to monitor the cluster and application so that I will know when the application/cluster is down. I cannot move to cluster deploy mode now. I appreciate your thoughts. Thanks -Anna
Logging in lSpark streaming application
Do I need to include the log4j dependencies in my pom.xml of the spark streaming application or it is already included in spark libraries? I am running Spark in standalone mode on AWS EC2. Thanks
Re: Spark streaming, Storage tab questions
On Sun, Jul 9, 2017 at 4:33 PM, anna stax wrote: > Does each row represent the state of my app at different time? > > When the fraction cached is 90% and the size on Disk is 0, does that mean > 10% of the data is lost. Its neither in memory now disk? > > I am running spark streaming standalone mode, and I am using mapWithState. > It runs fine for several hours and then fails. So trying to debug what is > causing it to fail. > > Please share your thoughts and experiences. > > Thanks > >
Spark streaming for CEP
Hello all, Has anyone used spark streaming for CEP (Complex Event processing). Any CEP libraries that works well with spark. I have a use case for CEP and trying to see if spark streaming is a good fit. Currently we have a data pipeline using Kafka, Spark streaming and Cassandra for data ingestion and near real time dashboard. Please share your experience. Thanks much. -Anna
Re: Spark streaming for CEP
Thanks very much Mich, Thomas and Stephan . I will look into it. On Tue, Oct 24, 2017 at 8:02 PM, lucas.g...@gmail.com wrote: > This looks really interesting, thanks for linking! > > Gary Lucas > > On 24 October 2017 at 15:06, Mich Talebzadeh > wrote: > >> Great thanks Steve >> >> Dr Mich Talebzadeh >> >> >> >> LinkedIn * >> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >> >> >> >> http://talebzadehmich.wordpress.com >> >> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for >> any loss, damage or destruction of data or any other property which may >> arise from relying on this email's technical content is explicitly >> disclaimed. The author will in no case be liable for any monetary damages >> arising from such loss, damage or destruction. >> >> >> >> On 24 October 2017 at 22:58, Stephen Boesch wrote: >> >>> Hi Mich, the github link has a brief intro - including a link to the >>> formal docs http://logisland.readthedocs.io/en/latest/index.html . >>> They have an architectural overview, developer guide, tutorial, and pretty >>> comprehensive api docs. >>> >>> 2017-10-24 13:31 GMT-07:00 Mich Talebzadeh : >>> >>>> thanks Thomas. >>>> >>>> do you have a summary write-up for this tool please? >>>> >>>> >>>> regards, >>>> >>>> >>>> >>>> >>>> Thomas >>>> >>>> Dr Mich Talebzadeh >>>> >>>> >>>> >>>> LinkedIn * >>>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>> <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>> >>>> >>>> >>>> http://talebzadehmich.wordpress.com >>>> >>>> >>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>>> any loss, damage or destruction of data or any other property which may >>>> arise from relying on this email's technical content is explicitly >>>> disclaimed. The author will in no case be liable for any monetary damages >>>> arising from such loss, damage or destruction. >>>> >>>> >>>> >>>> On 24 October 2017 at 13:53, Thomas Bailet >>>> wrote: >>>> >>>>> Hi >>>>> >>>>> we (@ hurence) have released on open source middleware based on >>>>> SparkStreaming over Kafka to do CEP and log mining, called *logisland* >>>>> (https://github.com/Hurence/logisland/) it has been deployed into >>>>> production for 2 years now and does a great job. You should have a look. >>>>> >>>>> >>>>> bye >>>>> >>>>> Thomas Bailet >>>>> >>>>> CTO : hurence >>>>> >>>>> Le 18/10/17 à 22:05, Mich Talebzadeh a écrit : >>>>> >>>>> As you may be aware the granularity that Spark streaming has is >>>>> micro-batching and that is limited to 0.5 second. So if you have >>>>> continuous >>>>> ingestion of data then Spark streaming may not be granular enough for CEP. >>>>> You may consider other products. >>>>> >>>>> Worth looking at this old thread on mine "Spark support for Complex >>>>> Event Processing (CEP) >>>>> >>>>> https://mail-archives.apache.org/mod_mbox/spark-user/201604. >>>>> mbox/%3CCAJ3fcbB8eaf0JV84bA7XGUK5GajC1yGT3ZgTNCi8arJg56=LbQ@ >>>>> mail.gmail.com%3E >>>>> >>>>> HTH >>>>> >>>>> >>>>> Dr Mich Talebzadeh >>>>> >>>>> >>>>> >>>>> LinkedIn * >>>>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>>> <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>>> >>>>> >>>>> >>>>> http://talebzadehmich.wordpress.com >>>>> >>>>> >>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>>>> any loss, damage or destruction of data or any other property which may >>>>> arise from relying on this email's technical content is explicitly >>>>> disclaimed. The author will in no case be liable for any monetary damages >>>>> arising from such loss, damage or destruction. >>>>> >>>>> >>>>> >>>>> On 18 October 2017 at 20:52, anna stax wrote: >>>>> >>>>>> Hello all, >>>>>> >>>>>> Has anyone used spark streaming for CEP (Complex Event processing). >>>>>> Any CEP libraries that works well with spark. I have a use case for CEP >>>>>> and >>>>>> trying to see if spark streaming is a good fit. >>>>>> >>>>>> Currently we have a data pipeline using Kafka, Spark streaming and >>>>>> Cassandra for data ingestion and near real time dashboard. >>>>>> >>>>>> Please share your experience. >>>>>> Thanks much. >>>>>> -Anna >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>> >> >
Re: How do I save the dataframe data as a pdf file?
Thanks Anthony for the response. Yes, the data in the dataframe represents a report and I want to create pdf files. I am using scala so hoping to find a easier solution in scala, if not I will try out your suggestion . On Tue, Dec 12, 2017 at 11:29 AM, Anthony Thomas wrote: > Are you trying to produce a formatted table in a pdf file where the > numbers in the table come from a dataframe? I.e. to present summary > statistics or other aggregates? If so I would guess your best bet would be > to collect the dataframe as a Pandas dataframe and use the to_latex method. > You can then use a standard latex compiler to produce a pdf with a table > containing that data. I don't know if there's any comparable built-in for > Scala, but you could always collect the data as an array of arrays and > write these to a tex file using standard IO. Maybe someone has an easier > suggestion. > > On Tue, Dec 12, 2017 at 11:12 AM, shyla deshpande < > deshpandesh...@gmail.com> wrote: > >> Hello all, >> >> Is there a way to write the dataframe data as a pdf file? >> >> Thanks >> -Shyla >> > >
Unable to see the table created using saveAsTable From Beeline. Please help!
I am running spark 2.1.0 on AWS EMR In my Zeppelin Note I am creating a table df.write .format("parquet") .saveAsTable("default.1test") and I see the table when I spark.catalog.listTables().show() +++---+-+---+ |name|database|description|tableType|isTemporary| +++---+-+---+ | 1test| default| null| MANAGED| false| >From Beeline client, I don’t see the table 0: jdbc:hive2://localhost:10001/> show tables; +---++--+--+ | database | tableName | isTemporary | +---++--+--+ +---++--+--+ No rows selected (0.115 seconds)
Re: Unable to see the table created using saveAsTable From Beeline. Please help!
Is some configuration missing ? Appreciate any help On Fri, Jul 6, 2018 at 4:10 PM, anna stax wrote: > I am running spark 2.1.0 on AWS EMR > > In my Zeppelin Note I am creating a table > > df.write > .format("parquet") > .saveAsTable("default.1test") > > and I see the table when I > > spark.catalog.listTables().show() > +++---+-+---+ > |name|database|description|tableType|isTemporary| > +++---+-+---+ > | 1test| default| null| MANAGED| false| > > > From Beeline client, I don’t see the table > > 0: jdbc:hive2://localhost:10001/> show tables; > +---++--+--+ > | database | tableName | isTemporary | > +---++--+--+ > +---++--+--+ > No rows selected (0.115 seconds) > >
Writing the contents of spark dataframe to Kafka with Spark 2.2
Hi all, I am unable to write the contents of spark dataframe to Kafka. I am using Spark 2.2 This is my code val df = Seq(("1","One"),("2","two")).toDF("key","value") df.printSchema() df.show(false) df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)") .write .format("kafka") .option("kafka.bootstrap.servers", "127.0.0.1:9092") .option("topic", "testtopic") .save() and I am getting the following error message [Stage 0:> (0 + 2) / 2]Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost, executor driver): java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.expressions.Cast$.apply$default$3()Lscala/Option; at org.apache.spark.sql.kafka010.KafkaWriteTask.createProjection(KafkaWriteTask.scala:112) at org.apache.spark.sql.kafka010.KafkaWriteTask.(KafkaWriteTask.scala:39) at org.apache.spark.sql.kafka010.KafkaWriter$$anonfun$write$1$$anonfun$apply$mcV$sp$1.apply(KafkaWriter.scala:90) at org.apache.spark.sql.kafka010.KafkaWriter$$anonfun$write$1$$anonfun$apply$mcV$sp$1.apply(KafkaWriter.scala:89) at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:926) at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:926) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1951) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1951) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:99) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) I have added this dependency org.apache.spark spark-sql_2.11 2.2.2 Appreciate any help. Thanks. https://stackoverflow.com/questions/55229945/writing-the-contents-of-spark-dataframe-to-kafka-with-spark-2-2
Re: Writing the contents of spark dataframe to Kafka with Spark 2.2
Hi Gabor, Thank you for the response. I do have those dependencies added. org.apache.spark spark-core_2.11 2.2.2 org.apache.spark spark-sql_2.11 2.2.2 org.apache.spark spark-sql-kafka-0-10_2.11 2.2.2 and my kafka version is kafka_2.11-1.1.0 On Tue, Mar 19, 2019 at 12:48 AM Gabor Somogyi wrote: > Hi Anna, > > Have you added spark-sql-kafka-0-10_2.11:2.2.0 package as well? > Further info can be found here: > https://spark.apache.org/docs/2.2.0/structured-streaming-kafka-integration.html#deploying > The same --packages option can be used with spark-shell as well... > > BR, > G > > > On Mon, Mar 18, 2019 at 10:07 PM anna stax wrote: > >> Hi all, >> I am unable to write the contents of spark dataframe to Kafka. >> I am using Spark 2.2 >> >> This is my code >> >> val df = Seq(("1","One"),("2","two")).toDF("key","value") >> df.printSchema() >> df.show(false) >> df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)") >> .write >> .format("kafka") >> .option("kafka.bootstrap.servers", "127.0.0.1:9092") >> .option("topic", "testtopic") >> .save() >> >> and I am getting the following error message >> [Stage 0:> (0 + >> 2) / 2]Exception in thread "main" org.apache.spark.SparkException: Job >> aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most >> recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost, executor >> driver): java.lang.NoSuchMethodError: >> org.apache.spark.sql.catalyst.expressions.Cast$.apply$default$3()Lscala/Option; >> at >> org.apache.spark.sql.kafka010.KafkaWriteTask.createProjection(KafkaWriteTask.scala:112) >> at >> org.apache.spark.sql.kafka010.KafkaWriteTask.(KafkaWriteTask.scala:39) >> at >> org.apache.spark.sql.kafka010.KafkaWriter$$anonfun$write$1$$anonfun$apply$mcV$sp$1.apply(KafkaWriter.scala:90) >> at >> org.apache.spark.sql.kafka010.KafkaWriter$$anonfun$write$1$$anonfun$apply$mcV$sp$1.apply(KafkaWriter.scala:89) >> at >> org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:926) >> at >> org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:926) >> at >> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1951) >> at >> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1951) >> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) >> at org.apache.spark.scheduler.Task.run(Task.scala:99) >> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> at java.lang.Thread.run(Thread.java:745) >> >> I have added this dependency >> >> >> org.apache.spark >> spark-sql_2.11 >> 2.2.2 >> >> >> Appreciate any help. Thanks. >> >> https://stackoverflow.com/questions/55229945/writing-the-contents-of-spark-dataframe-to-kafka-with-spark-2-2 >> >
Re: Writing the contents of spark dataframe to Kafka with Spark 2.2
Hi Gabor, I am just trying out. Desperately trying to make writing to kafka work using spark-sql-kafka. In my deployment project I do use the provided scope. On Tue, Mar 19, 2019 at 8:50 AM Gabor Somogyi wrote: > Hi Anna, > > Looks like some sort of version mismatch. > > Presume scala version double checked... > > Not sure why the mentioned artifacts are not in provided scope. > It will end-up in significantly smaller jar + these artifacts should be > available on the cluster (either by default for example core or due to > --packages for example the sql-kafka). > > BR, > G > > > On Tue, Mar 19, 2019 at 4:35 PM anna stax wrote: > >> Hi Gabor, >> >> Thank you for the response. >> >> I do have those dependencies added. >> >> >> org.apache.spark >> spark-core_2.11 >> 2.2.2 >> >> >> org.apache.spark >> spark-sql_2.11 >> 2.2.2 >> >> >> org.apache.spark >> spark-sql-kafka-0-10_2.11 >> 2.2.2 >> >> >> and my kafka version is kafka_2.11-1.1.0 >> >> >> On Tue, Mar 19, 2019 at 12:48 AM Gabor Somogyi >> wrote: >> >>> Hi Anna, >>> >>> Have you added spark-sql-kafka-0-10_2.11:2.2.0 package as well? >>> Further info can be found here: >>> https://spark.apache.org/docs/2.2.0/structured-streaming-kafka-integration.html#deploying >>> The same --packages option can be used with spark-shell as well... >>> >>> BR, >>> G >>> >>> >>> On Mon, Mar 18, 2019 at 10:07 PM anna stax wrote: >>> >>>> Hi all, >>>> I am unable to write the contents of spark dataframe to Kafka. >>>> I am using Spark 2.2 >>>> >>>> This is my code >>>> >>>> val df = Seq(("1","One"),("2","two")).toDF("key","value") >>>> df.printSchema() >>>> df.show(false) >>>> df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)") >>>> .write >>>> .format("kafka") >>>> .option("kafka.bootstrap.servers", "127.0.0.1:9092") >>>> .option("topic", "testtopic") >>>> .save() >>>> >>>> and I am getting the following error message >>>> [Stage 0:> (0 >>>> + 2) / 2]Exception in thread "main" org.apache.spark.SparkException: Job >>>> aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most >>>> recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost, executor >>>> driver): java.lang.NoSuchMethodError: >>>> org.apache.spark.sql.catalyst.expressions.Cast$.apply$default$3()Lscala/Option; >>>> at >>>> org.apache.spark.sql.kafka010.KafkaWriteTask.createProjection(KafkaWriteTask.scala:112) >>>> at >>>> org.apache.spark.sql.kafka010.KafkaWriteTask.(KafkaWriteTask.scala:39) >>>> at >>>> org.apache.spark.sql.kafka010.KafkaWriter$$anonfun$write$1$$anonfun$apply$mcV$sp$1.apply(KafkaWriter.scala:90) >>>> at >>>> org.apache.spark.sql.kafka010.KafkaWriter$$anonfun$write$1$$anonfun$apply$mcV$sp$1.apply(KafkaWriter.scala:89) >>>> at >>>> org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:926) >>>> at >>>> org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:926) >>>> at >>>> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1951) >>>> at >>>> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1951) >>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) >>>> at org.apache.spark.scheduler.Task.run(Task.scala:99) >>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>> at java.lang.Thread.run(Thread.java:745) >>>> >>>> I have added this dependency >>>> >>>> >>>> org.apache.spark >>>> spark-sql_2.11 >>>> 2.2.2 >>>> >>>> >>>> Appreciate any help. Thanks. >>>> >>>> https://stackoverflow.com/questions/55229945/writing-the-contents-of-spark-dataframe-to-kafka-with-spark-2-2 >>>> >>>