Re: ES ignores queries through Spark

2015-04-09 Thread Michael Czerny
Sorry about that! I am using 2.1.0.Beta3. I see. I tried using a date range instead, and that query seems to work, so I believe your answer is correct. Thanks! On Tuesday, April 7, 2015 at 4:34:43 PM UTC-4, Costin Leau wrote: You haven't specified the version of elasticsearch-spark. Either

Re: ES ignores queries through Spark

2015-04-07 Thread Costin Leau
You haven't specified the version of elasticsearch-spark. Either way the issue is likely caused by the fact that each query is executed locally on each target shard. In other words, your limit of 1000 entries is executed on each shard so at a maximum you will have 1000 x number of shards. By

ES ignores queries through Spark

2015-04-07 Thread Michael Czerny
Hi all, So I managed to get elasticsearch-spark_2.10 to work and I can query a database of tweets in Spark. The problem is it seems to ignore my specific queries, for example specifying the size or the fields to return. For example, this is my code: import org.apache.spark._ import