Is this about Spark SQL vs Redshift, or Spark in general? Spark in general provides a broader set of capabilities than Redshift because it has APIs in general-purpose languages (Java, Scala, Python) and libraries for things like machine learning and graph processing. For example, you might use Spark to do the ETL that will put data into a database such as Redshift, or you might pull data out of Redshift into Spark for machine learning. On the other hand, if *all* you want to do is SQL and you are okay with the set of data formats and features in Redshift (i.e. you can express everything using its UDFs and you have a way to get data in), then Redshift is a complete service which will do more management out of the box.
Matei > On Nov 4, 2014, at 3:11 PM, agfung <agf...@gmail.com> wrote: > > I'm in the midst of a heated debate about the use of Redshift v Spark with a > colleague. We keep trading anecdotes and links back and forth (eg airbnb > post from 2013 or amplab benchmarks), and we don't seem to be getting > anywhere. > > So before we start down the prototype /benchmark road, and in desperation > of finding *some* kind of objective third party perspective, was wondering > if anyone who has used both in 2014 would care to provide commentary about > the sweet spot use cases / gotchas for non trivial use (eg a simple filter > scan isn't really interesting). Soft issues like operational maintenance > and time spent developing v out of the box are interesting too... > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-v-Redshift-tp18112.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
