Re: is it ok to have multiple sparksession's in one spark structured streaming app?
You would set the Kafka topic as your data source and you would write a custom output to Cassandra everything would be or could be contained within your stream -Paul Sent from my iPhone > On Sep 8, 2017, at 2:52 PM, kant kodali <kanth...@gmail.com> wrote: > > How can I use one SparkSession to talk to both Kafka and Cassandra let's say? > > >> On Fri, Sep 8, 2017 at 3:46 AM, Arkadiusz Bicz <arkadiusz.b...@gmail.com> >> wrote: >> You don't need multiple spark sessions to have more than one stream working, >> but from maintenance and reliability perspective it is not good idea. >> >>> On Thu, Sep 7, 2017 at 2:40 AM, kant kodali <kanth...@gmail.com> wrote: >>> Hi All, >>> >>> I am wondering if it is ok to have multiple sparksession's in one spark >>> structured streaming app? Basically, I want to create 1) Spark session for >>> reading from Kafka and 2) Another Spark session for storing the mutations >>> of a dataframe/dataset to a persistent table as I get the mutations from >>> #1? >>> >>> Finally, is this a common practice? >>> >>> Thanks, >>> kant >> >
Re: is it ok to have multiple sparksession's in one spark structured streaming app?
How can I use one SparkSession to talk to both Kafka and Cassandra let's say? On Fri, Sep 8, 2017 at 3:46 AM, Arkadiusz Bicz <arkadiusz.b...@gmail.com> wrote: > You don't need multiple spark sessions to have more than one stream > working, but from maintenance and reliability perspective it is not good > idea. > > On Thu, Sep 7, 2017 at 2:40 AM, kant kodali <kanth...@gmail.com> wrote: > >> Hi All, >> >> I am wondering if it is ok to have multiple sparksession's in one spark >> structured streaming app? Basically, I want to create 1) Spark session for >> reading from Kafka and 2) Another Spark session for storing the mutations >> of a dataframe/dataset to a persistent table as I get the mutations from >> #1? >> >> Finally, is this a common practice? >> >> Thanks, >> kant >> > >
Re: is it ok to have multiple sparksession's in one spark structured streaming app?
You don't need multiple spark sessions to have more than one stream working, but from maintenance and reliability perspective it is not good idea. On Thu, Sep 7, 2017 at 2:40 AM, kant kodali <kanth...@gmail.com> wrote: > Hi All, > > I am wondering if it is ok to have multiple sparksession's in one spark > structured streaming app? Basically, I want to create 1) Spark session for > reading from Kafka and 2) Another Spark session for storing the mutations > of a dataframe/dataset to a persistent table as I get the mutations from > #1? > > Finally, is this a common practice? > > Thanks, > kant >
is it ok to have multiple sparksession's in one spark structured streaming app?
Hi All, I am wondering if it is ok to have multiple sparksession's in one spark structured streaming app? Basically, I want to create 1) Spark session for reading from Kafka and 2) Another Spark session for storing the mutations of a dataframe/dataset to a persistent table as I get the mutations from #1? Finally, is this a common practice? Thanks, kant
