Hi Ayan,How "continuous" is your workload? As Akhil points out, with streaming,
you'll give up at least one core for receiving, will need at most one more core
for processing. Unless you're running on something like Mesos, this means that
those cores are dedicated to your app, and can't be leveraged by other apps /
jobs.
If it's something periodic (once an hour, once every 15 minutes, etc.), then
I'd simply write a "normal" spark application, and trigger it periodically.
There are many things that can take care of that - sometimes a simple cronjob
is enough!
Date: Sun, 5 Jul 2015 22:48:37 +1000
Subject: Re: JDBC Streams
From: guha.a...@gmail.com
To: ak...@sigmoidanalytics.com
CC: user@spark.apache.org
Thanks Akhil. In case I go with spark streaming, I guess I have to implment a
custom receiver and spark streaming will call this receiver every batch
interval, is that correct? Any gotcha you see in this plan? TIA...Best, Ayan
On Sun, Jul 5, 2015 at 5:40 PM, Akhil Das <ak...@sigmoidanalytics.com> wrote:
If you want a long running application, then go with spark streaming (which
kind of blocks your resources). On the other hand, if you use job server then
you can actually use the resources (CPUs) for other jobs also when your dbjob
is not using them.ThanksBest Regards
On Sun, Jul 5, 2015 at 5:28 AM, ayan guha <guha.a...@gmail.com> wrote:
Hi All
I have a requireent to connect to a DB every few minutes and bring data to
HBase. Can anyone suggest if spark streaming would be appropriate for this
senario or I shoud look into jobserver?
Thanks in advance
--
Best Regards,
Ayan Guha
--
Best Regards,
Ayan Guha
