I haven’t heard of Kafka connect. I’ll have to look into it. Kafka would, of course have to be in any architecture but it looks like they are suggesting that Kafka is all you need.
My primary concern is the complexity of loading warehouses. I have a web development background so I have somewhat of an idea on how to insert data into a database from an application. I’ve since moved on to straight database programming and don’t work with anything that reads from an app anymore. Loading a warehouse requires a lot of cleaning of data and running and grabbing keys to maintain referential integrity. Usually that’s done in a batch process. Now I have to do it record by record (or a few records). I have some ideas but I’m not quite there yet. I thought SparkSQL would be the way to get this done but so far, all the examples I’ve seen are just SELECT statements, no INSERTS or MERGE statements. Adaryl "Bob" Wakefield, MBA Principal Mass Street Analytics, LLC 913.938.6685 www.massstreet.net<http://www.massstreet.net> www.linkedin.com/in/bobwakefieldmba<http://www.linkedin.com/in/bobwakefieldmba> Twitter: @BobLovesData From: Femi Anthony [mailto:femib...@gmail.com] Sent: Tuesday, February 28, 2017 4:13 AM To: Adaryl Wakefield <adaryl.wakefi...@hotmail.com> Cc: user@spark.apache.org Subject: Re: using spark to load a data warehouse in real time Have you checked to see if there are any drivers to enable you to write to Greenplum directly from Spark ? You can also take a look at this link: https://groups.google.com/a/greenplum.org/forum/m/#!topic/gpdb-users/lnm0Z7WBW6Q Apparently GPDB is based on Postgres so maybe that approach may work. Another approach maybe for Spark Streaming to write to Kafka, and then have another process read from Kafka and write to Greenplum. Kafka Connect may be useful in this case - https://www.confluent.io/blog/announcing-kafka-connect-building-large-scale-low-latency-data-pipelines/ Femi Anthony On Feb 27, 2017, at 7:18 PM, Adaryl Wakefield <adaryl.wakefi...@hotmail.com<mailto:adaryl.wakefi...@hotmail.com>> wrote: Is anybody using Spark streaming/SQL to load a relational data warehouse in real time? There isn’t a lot of information on this use case out there. When I google real time data warehouse load, nothing I find is up to date. It’s all turn of the century stuff and doesn’t take into account advancements in database technology. Additionally, whenever I try to learn spark, it’s always the same thing. Play with twitter data never structured data. All the CEP uses cases are about data science. I’d like to use Spark to load Greenplumb in real time. Intuitively, this should be possible. I was thinking Spark Streaming with Spark SQL along with a ORM should do it. Am I off base with this? Is the reason why there are no examples is because there is a better way to do what I want? Adaryl "Bob" Wakefield, MBA Principal Mass Street Analytics, LLC 913.938.6685 www.massstreet.net<http://www.massstreet.net> www.linkedin.com/in/bobwakefieldmba<http://www.linkedin.com/in/bobwakefieldmba> Twitter: @BobLovesData