Re: Is Spark suited for replacing a batch job using many database tables?

2016-07-06 Thread Andreas Bauer
, assuming you have the appropriate indexes in place. You could also look at Apache Drill too. HTH -Mike > On Jul 6, 2016, at 3:24 PM, Andreas Bauer <dabuks...@gmail.com> wrote: > > Thanks for the advice. I have to retrieve the basic data from the DB2 tables >

Re: Is Spark suited for replacing a batch job using many database tables?

2016-07-06 Thread Andreas Bauer
Thanks for the advice. I have to retrieve the basic data from the DB2 tables but afterwards I'm pretty free to transform the data as needed.  I think you need to learn the basics of how to build a ‘data lake/pond/sewer’ first. The short answer is yes. The longer answer is that you need

Re: Is Spark suited for replacing a batch job using many database tables?

2016-07-06 Thread Andreas Bauer
uction.   On 6 July 2016 at 20:54, Andreas Bauer <dabuks...@gmail.com> wrote: In fact, yes. On 6. Juli 2016 um 21:46:34 MESZ, Mich Talebzadeh <mich.talebza...@gmail.com> wrote:So you want to use Spark as the query engine accessing DB2 tables via JDBC? Dr Mich Talebzade

Re: Is Spark suited for replacing a batch job using many database tables?

2016-07-06 Thread Andreas Bauer
author will in no case be liable for any monetary damages arising from such loss, damage or destruction.   On 6 July 2016 at 20:39, Andreas Bauer <dabuks...@gmail.com> wrote:The sql statements are embedded in a PL/1 program using DB2 running ob z/OS. Quite powerful, but expensive and for

Re: Is Spark suited for replacing a batch job using many database tables?

2016-07-06 Thread Andreas Bauer
The sql statements are embedded in a PL/1 program using DB2 running ob z/OS. Quite powerful, but expensive and foremost shared withother jobs in the comapny. The whole job takes approx. 20 minutes.  So I was thinking to use Spark and let the Spark job run on 10 or 20 virtual instances, which I