[DISCUSS] Abstracting away Hadoop/MapReduce as Data Processing Layer

2014-07-07 Thread Lewis John Mcgibbney
Hi Folks, Many people know the way that things are going with regards to in-memory computing being 'the' hot topic on the planet right now (outside of the world cup). We have made good strides in Gora to get it to where it is as a top level project. It has also become aparent to me that something w

Re: [DISCUSS] Abstracting away Hadoop/MapReduce as Data Processing Layer

2014-07-09 Thread Henry Saputra
Internally, Apache Spark can use Hadoop input format for its distributed data structure (a.k.a RDD). So, I guess we could still join the cool kids with Spark via our input format implementation. However, I could think of other improvements that could be useful (apology to Lewis if I hijacked his d

Re: [DISCUSS] Abstracting away Hadoop/MapReduce as Data Processing Layer

2014-07-09 Thread Renato MarroquĂ­n Mogrovejo
2014-07-09 11:10 GMT+02:00 Henry Saputra : > Internally, Apache Spark can use Hadoop input format for its > distributed data structure (a.k.a RDD). > So, I guess we could still join the cool kids with Spark via our input > format implementation. > Cool Henry! I didn't know about we could use Hado

Re: [DISCUSS] Abstracting away Hadoop/MapReduce as Data Processing Layer

2014-07-09 Thread Lewis John Mcgibbney
Hi Henry, On Wed, Jul 9, 2014 at 5:10 AM, wrote: > > However, I could think of other improvements that could be useful > (apology to Lewis if I hijacked his discussion): > 1. Pluggable serialization mechanism to allow other like Thrift or > Protocol Buffer instead of just Avro. > 2. Directly wor

Re: [DISCUSS] Abstracting away Hadoop/MapReduce as Data Processing Layer

2014-07-09 Thread Henry Saputra
For pluggable serialization, I think if there is not JIRA opened I could open one as reccomended by Lewis. As for low hanging fruit, I am currently not sure. Maybe we could add Gora store manager to Spark to allow read and persist from different NoSQL databases. - Henry On Wed, Jul 9, 2014 at 2:

Re: [DISCUSS] Abstracting away Hadoop/MapReduce as Data Processing Layer

2014-07-09 Thread Henry Saputra
HI Lewis, yeah I could file issues so we could track them for next releases Thanks for starting discussion about new innovation with Gora. +1 - Henry On Wed, Jul 9, 2014 at 3:51 AM, Lewis John Mcgibbney wrote: > Hi Henry, > > On Wed, Jul 9, 2014 at 5:10 AM, wrote: > >> >> However, I could thi