If your data only changes every few days, why not restart the job every few days, and just broadcast the data?
Or you can keep a local per-jvm cache with an expiry (e.g. guava cache) to avoid many mysql reads On Wed, Aug 26, 2015 at 9:46 AM, Chen Song <chen.song...@gmail.com> wrote: > Piggyback on this question. > > I have a similar use case but a bit different. My job is consuming a > stream from Kafka and I need to join the Kafka stream with some reference > table from MySQL (kind of data validation and enrichment). I need to > process this stream every 1 min. The data in MySQL is not changed very > often, maybe once a few days. > > So my requirement is: > > * I cannot easily use broadcast variable because the data does change, > although not very often. > * I am not sure if it is good practice to read data from MySQL in every > batch (in my case, 1 min). > > Anyone has done this before, any suggestions and feedback is appreciated. > > Chen > > > On Sun, Jul 5, 2015 at 11:50 AM, Ashic Mahtab <as...@live.com> wrote: > >> If it is indeed a reactive use case, then Spark Streaming would be a good >> choice. >> >> One approach worth considering - is it possible to receive a message via >> kafka (or some other queue). That'd not need any polling, and you could use >> standard consumers. If polling isn't an issue, then writing a custom >> receiver will work fine. The way a receiver works is this: >> >> * Your receiver has a receive() function, where you'd typically start a >> loop. In your loop, you'd fetch items, and call store(entry). >> * You control everything in the receiver. If you're listening on a queue, >> you receive messages, store() and ack your queue. If you're polling, it's >> up to you to ensure delays between db calls. >> * The things you store() go on to make up the rdds in your DStream. So, >> intervals, windowing, etc. apply to those. The receiver is the boundary >> between your data source and the DStream RDDs. In other words, if your >> interval is 15 seconds with no windowing, then the things that went to >> store() every 15 seconds are bunched up into an RDD of your DStream. That's >> kind of a simplification, but should give you the idea that your "db >> polling" interval and streaming interval are not tied together. >> >> -Ashic. >> >> ------------------------------ >> Date: Mon, 6 Jul 2015 01:12:34 +1000 >> Subject: Re: JDBC Streams >> From: guha.a...@gmail.com >> To: as...@live.com >> CC: ak...@sigmoidanalytics.com; user@spark.apache.org >> >> >> Hi >> >> Thanks for the reply. here is my situation: I hve a DB which enbles >> synchronus CDC, think this as a DBtrigger which writes to a taable with >> "changed" values as soon as something changes in production table. My job >> will need to pick up the data "as soon as it arrives" which can be every 1 >> min interval. Ideally it will pick up the changes, transform it into a >> jsonand puts it to kinesis. In short, I am emulating a Kinesis producer >> with a DB source (dont even ask why, lets say these are the constraints :) ) >> >> Please advice (a) is spark a good choice here (b) whats your suggestion >> either way. >> >> I understand I can easily do it using a simple java/python app but I am >> little worried about managing scaling/fault tolerance and thats where my >> concern is. >> >> TIA >> Ayan >> >> On Mon, Jul 6, 2015 at 12:51 AM, Ashic Mahtab <as...@live.com> wrote: >> >> Hi Ayan, >> How "continuous" is your workload? As Akhil points out, with streaming, >> you'll give up at least one core for receiving, will need at most one more >> core for processing. Unless you're running on something like Mesos, this >> means that those cores are dedicated to your app, and can't be leveraged by >> other apps / jobs. >> >> If it's something periodic (once an hour, once every 15 minutes, etc.), >> then I'd simply write a "normal" spark application, and trigger it >> periodically. There are many things that can take care of that - sometimes >> a simple cronjob is enough! >> >> ------------------------------ >> Date: Sun, 5 Jul 2015 22:48:37 +1000 >> Subject: Re: JDBC Streams >> From: guha.a...@gmail.com >> To: ak...@sigmoidanalytics.com >> CC: user@spark.apache.org >> >> >> Thanks Akhil. In case I go with spark streaming, I guess I have to >> implment a custom receiver and spark streaming will call this receiver >> every batch interval, is that correct? Any gotcha you see in this plan? >> TIA...Best, Ayan >> >> On Sun, Jul 5, 2015 at 5:40 PM, Akhil Das <ak...@sigmoidanalytics.com> >> wrote: >> >> If you want a long running application, then go with spark streaming >> (which kind of blocks your resources). On the other hand, if you use job >> server then you can actually use the resources (CPUs) for other jobs also >> when your dbjob is not using them. >> >> Thanks >> Best Regards >> >> On Sun, Jul 5, 2015 at 5:28 AM, ayan guha <guha.a...@gmail.com> wrote: >> >> Hi All >> >> I have a requireent to connect to a DB every few minutes and bring data >> to HBase. Can anyone suggest if spark streaming would be appropriate for >> this senario or I shoud look into jobserver? >> >> Thanks in advance >> >> -- >> Best Regards, >> Ayan Guha >> >> >> >> >> >> -- >> Best Regards, >> Ayan Guha >> >> >> >> >> -- >> Best Regards, >> Ayan Guha >> > > > > -- > Chen Song > >
