Github user jeffsteinmetz commented on the pull request: https://github.com/apache/incubator-zeppelin/pull/520#issuecomment-162749562 I love the idea of making elasticsearch a first class cititen to Zeppelin. I was curious however, as you build out the query language, eventually it will get to the point that you'll want to add more features to keep in parity with elasticsearch's existing "extensive" query dsl. I've built some scala libraries that wrap ES for usage in API, spark utilities and other business logic. Eventually the wrapper even starts to try to mimic some of the ES query language. As a long time user of Elasticsearch, elasticsearch-hadoop (spark integration), beta testing elasticsearch-hadoop, using ES from spark, etc, my one comment is the construction of another DSL (domain specific language) via an interpreter. Curious if there has been any considerations made as to why you might not just pass through existing ES json using the existing query language? Imagine if we abstracted another query language on top of Spark SQL %sql that wasn't SQL? The ease of use is certainly welcome though. So I was curious about the long term plans for the DSL in the interpreter. As a side note, I import this library, https://github.com/sksamuel/elastic4s when I want to simplify my Elastic Search query experience ( this DSL misses out on a few functional builder patterns and recent features, but does cover about 80% of most use cases): Using the native spark integration provided by Elasticsearch Hadoop after an `import org.elasticsearch.spark._` , you have 2 options to load the results into an RDD, either the original json: `val myRDD: RDD[(String, String)] = sc.esJsonRDD("someindex/doctype", query)` or a Map `val myRDD: RDD[(String, String)] = sc.esRDD("someindex/doctype", query)` Elasticsearch Hadoop (spark) can also create a dataframe using the elasticsearch spark sql context. Curious if there has been any thought around how the elasticsearch interpreter / pipeline could be a bit more spark, ES native and ES hadoop native friendly.
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---