Yan, it looks like Julian did anticipate exactly this possibility: https://github.com/julianhyde/optiq/tree/master/spark
Optiq is a cool project vision in terms of hiding various engines behind one consistent API. That said, from just the Spark perspective, I don't see a huge value add to layer Optiq above SparkSQL---until and unless Optiq provides a lot more idioms and/or operational facilities than just making Spark RDDs look like tables, which SparkSQL already does quite nicely and increasingly. Warehousing, perhaps? Here, I can't avoid a mention of DDF which aims to add more algorithmic and data manipulation value in addition to the table abstraction ( https://spark-summit.org/2014/talk/distributed-dataframe-ddf-on-apache-spark-simplifying-big-data-for-the-rest-of-us ) -- Christopher T. Nguyen Co-founder & CEO, Adatao <http://adatao.com> linkedin.com/in/ctnguyen On Fri, Jun 6, 2014 at 11:26 AM, Yan Zhou.sc <yan.zhou...@huawei.com> wrote: > Can anybody share your thoughts/comments/interests of applicability of the > "optiq" framework to Spark, and SparkSQL in particular? > > Thanks, >