Re: Questions about dedicated master & client node

[James Macdonald]

If it is good enough for you, it is good enough for you. I will just give
you one anecdote: We implemented 3 dedicated clients on a 9 data node
cluster and got a 2x performance improvement. Moving the query
coordination, network io (has to receive data from every shard), and
combination of results (aggs and sorts) off of the nodes providing the
results is very helpful.

James



On Sat, May 30, 2015 at 9:11 AM, Xudong You  wrote:

> Thanks Nikolas,
> How do you think about dedicated "client node" (the so called load balance
> node)? Any benefit of dedicated client node? Seems to me, round robin to
> data nodes is good enough.
>
> On Friday, May 29, 2015 at 10:55:01 PM UTC+8, Nikolas Everett wrote:
>>
>> Dedicated master nodes are super convenient if you have the it
>> infrastructure to host them on shared machines because they are very low
>> load and its useful to be able to restart the master nodes quickly. We
>> don't have that kind of infrastructure and our cluster is pretty large and
>> not having it has bitten us once or twice but its not a huge problem.
>>
>>
>>
>> On Fri, May 29, 2015 at 10:44 AM, Xudong You  wrote:
>>
>>> Right now we only need 4 ES nodes due to the small data volume, and all
>>> 4 nodes are master & data nodes.
>>>
>>> Q1:
>>> I am wondering in this case, is it necessary to have dedicated master
>>> and client node? Any benefit of having dedicated master node?
>>>
>>> Some one said that dedicated master nodes (say, three master nodes) is
>>> helpful to avoid the split brain issue, but even we have NO dedicated
>>> master nodes, we can also avoid the split brain by setting the 
>>> *discovery.zen.minimum_master_nodes
>>> *to a appropriate value.
>>>
>>> Q2:
>>> Similarly, is dedicated client node really necessary in our 4 nodes
>>> case? Any benefit of allocating dedicated client node?
>>>
>>> Thanks!
>>>
>>> --
>>> Please update your bookmarks! We have moved to
>>> https://discuss.elastic.co/
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elasticsearch/de7db788-a6d2-48c2-934b-bc5f7ae311a9%40googlegroups.com
>>> <https://groups.google.com/d/msgid/elasticsearch/de7db788-a6d2-48c2-934b-bc5f7ae311a9%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> Please update your bookmarks! We have moved to https://discuss.elastic.co/
> ---
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/d61afae2-2e47-4b65-866b-5a55d28b84ea%40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/d61afae2-2e47-4b65-866b-5a55d28b84ea%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAABsnTbQispJSH%3D7_wbk-W5%2BmMq1_4Yy2Mxeh8RL%2BAAYeaRx6g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Aggregation profiling?

[James Macdonald]

I don't have an answer, but I really like this question. I too would love
to see more query and aggregation profiling tools for performance
optimization purposes.

Also, I assume you have already looked at this, but have you made sure you
are not evicting anything from your in memory field data?

James

On Mon, May 25, 2015 at 4:08 PM, Mike Sukmanowsky <
mike.sukmanow...@gmail.com> wrote:

> I don't believe there are any current endpoints in the API that support
> this, but are there plans to add better profiling information to ES
> aggregation queries? We'll see some agg queries return in 11s, then <5s
> then >11s again. Sometimes we can see associated filter cache expirations,
> but it's really hard to line these up to one specific query in our
> production environment since multiple users are executing queries
> simultaneously.
>
> It'd be really helpful to optionally see where aggregation queries are
> spending the bulk of their time to help us understand what to improve in
> the future.
>
> Anything we can do here right now?
>
> --
> Mike Sukmanowsky
> Aspiring Digital Carpenter
>
> *e*: mike.sukmanow...@gmail.com
>
> facebook <http://facebook.com/mike.sukmanowsky> | twitter
> <http://twitter.com/msukmanowsky> | LinkedIn
> <http://www.linkedin.com/profile/view?id=10897143> | github
> <https://github.com/msukmanowsky>
>
>   --
> Please update your bookmarks! We have moved to https://discuss.elastic.co/
> ---
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAOH6cu5WSGqQ%2BZ0_qrofXEvwo8JuSH9xoSbZgSwiT90MJ_wxdA%40mail.gmail.com
> <https://groups.google.com/d/msgid/elasticsearch/CAOH6cu5WSGqQ%2BZ0_qrofXEvwo8JuSH9xoSbZgSwiT90MJ_wxdA%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAABsnTZOmx-fk%2BG9dR6-XYB_1j7mGRNRwTqvQRwKx0YAcopFWA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Shard query cache size

[James Macdonald]

Hi Adrien, thanks very much for this clarification.  I am always trying to
learn more about how Elasticsearch works, and that clarification was very
helpful.

James

On Thu, May 21, 2015 at 6:12 PM, Adrien Grand  wrote:

> On Thu, May 21, 2015 at 11:49 PM, James Macdonald <
> james.macdon...@geofeedia.com> wrote:
>
>> Hi, I am a little confused by your response. Are you saying that
>> query/filter caches are invalidated across all data in a shard every time
>> the refresh interval ticks over?
>>
>
> Sorry for the confusion:
>  - the query cache caches entire requests per index, and is competely
> invalidated across all data every time the refresh interval ticks over AND
> there have been changes since the last refresh
>  - the filter cache caches matching documents per segment, it is
> invalidated per segment only when a segment goes away (typically because
> it's been merged to a larger segment), which is unfrequent for large
> segments
>  - the fielddata cache caches the document->value mapping per segment and
> has the same invalidation rules as the filter cache
>
>
>> I was under the impression that all field data and caching related
>> operations were performed on a Lucene index segment level and that the
>> caches would only be invalidated for a given segment if that segment had
>> changed since the last refresh. Since most data is stored in large segments
>> that don't take fresh writes and seldom merge this would mean that most
>> caches are good for long periods of time; even if the shard is under
>> constant indexing load. Am I mistaken?
>>
>
> This is right for the fielddata and filter caches, but not for the query
> cache.
>
> --
> Adrien
>
> --
> Please update your bookmarks! We have moved to https://discuss.elastic.co/
> ---
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAO5%3DkAjZBXouCRxNtvVmtsx%2BZK1_iymRC-7YSojULF7pSK%2Bebg%40mail.gmail.com
> <https://groups.google.com/d/msgid/elasticsearch/CAO5%3DkAjZBXouCRxNtvVmtsx%2BZK1_iymRC-7YSojULF7pSK%2Bebg%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAABsnTZtOWEF%3DRjBj_Xmt9x%2BHQdREn4vT2dFnbjRiF88oPnofA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Shard query cache size

[James Macdonald]

Hi, I am a little confused by your response. Are you saying that
query/filter caches are invalidated across all data in a shard every time
the refresh interval ticks over?

I was under the impression that all field data and caching related
operations were performed on a Lucene index segment level and that the
caches would only be invalidated for a given segment if that segment had
changed since the last refresh. Since most data is stored in large segments
that don't take fresh writes and seldom merge this would mean that most
caches are good for long periods of time; even if the shard is under
constant indexing load. Am I mistaken?

Thanks,
James

On Thu, May 21, 2015 at 9:28 AM, Adrien Grand  wrote:

> It depends how likely it is for you to run the same aggregation again.
> Note that this cache is fully invalidated at every refresh (meaning either
> every second by default, or every time that you update/add/remove documents
> if you perform less than 1 operation per second). So this cache will only
> be used if you are likely to run the exact same request twice or more in a
> short period of time.
>
> We assumed that this situation is not common, so a small cache would be
> enough for the maybe 4 or 5 requests that would be run again and again. You
> can increase it if you think it will be helpful in your case although I
> would advise to be careful, maybe memory would be better spent on eg. the
> filesystem cache.
>
> On Thu, May 21, 2015 at 3:41 PM, Mike Sukmanowsky <
> mike.sukmanow...@gmail.com> wrote:
>
>> Hi all,
>>
>> We store Marvel-style timeseries data in Elasticsearch and make very
>> heavy use of aggregations (all queries are effectively aggregations).
>>
>> We've been playing around with the shard query cache and have a question.
>>
>> Is there a reason the shard query cache is set to such a low level of JVM
>> heap by default? 1% seems awfully low unless ES assumes most people aren't
>> making heavy use of aggregations? Any harm in us significantly boosting
>> this from 1% to say 15% of heap? Most of our machines have 30GB of RAM and
>> heap at 50% of that (15GB) so the query cache is 150MB by default. Think
>> we'd like to experiment growing that to at least 10% of heap to have 1GB in
>> use for this cache.
>>
>> Mike
>>
>> --
>> Please update your bookmarks! We have moved to
>> https://discuss.elastic.co/
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/a3a0fa7b-49f8-4d78-a520-6eeb16d53de3%40googlegroups.com
>> <https://groups.google.com/d/msgid/elasticsearch/a3a0fa7b-49f8-4d78-a520-6eeb16d53de3%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> Adrien
>
> --
> Please update your bookmarks! We have moved to https://discuss.elastic.co/
> ---
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAO5%3DkAhjZW%2B7BoLqXOk8nEfOvaKrgUroZhbG41BVUDh2bHhEnQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/elasticsearch/CAO5%3DkAhjZW%2B7BoLqXOk8nEfOvaKrgUroZhbG41BVUDh2bHhEnQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAABsnTYY85YeKRUBFLoMcAfnPj%3DBP6GqM020N%2BrZ9yp32%2BQY0A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Feedback on Document Structure (and Inefficiencies)

[James Addison]

I'm looking to see if I can structure my current document schema better for 
the functionality we're providing - or maybe it's the queries that need 
improvement. Everything is working fine, but I can't help but see 
inefficiencies.  Keep in mind that this is based on relational data from a 
Django project.

The basic idea is we have two types of Activities: Events and Classes.

An Event is a much simplified version of a Class - it's essentially a 
series of dates, each with a start and end time. The dates are usually 
sequential and while the times are usually the same, they can differ (the 
last day in the Event's date sequence might end earlier, for example).

A Class is more complex:

   - each Class has one or more Sessions
   - each Session has one or more 'session_data' structures, which 
   holds days of the week (represented by a list of numbers, from 0-6), start 
   and end dates, and start/end times
   - reason for this is that a Session might happen from May 6 to Sept 8 on 
  Tuesdays at 8pm and Thursdays at 7:30pm - two 'session_data' structures 
are 
  needed to maintain this
   

Each Event and Class is indexed in Elasticsearch with the same document 
mapping 'activity' with a 'type' field. This is trimmed for brevity:

{
"properties": {
"type": {
"type": "long"
},
"name": {
"type": "string"
},
"dates": {
"type": "date",
"format": "dateOptionalTime"
},
"sessions": {
"properties": {
"session_data": {
"properties": {
"days_of_week": {
"type": "long"
},
"end": {
"type": "date",
"format": "dateOptionalTime"
},
"end_time": {
"type": "date",
"format": "hour_minute"
},
"start": {
"type": "date",
"format": "dateOptionalTime"
},
"start_time": {
"type": "date",
"format": "hour_minute"
}
}
},
"end": {
"type": "date",
"format": "dateOptionalTime"
},
"start": {
"type": "date",
"format": "dateOptionalTime"
}
}
}
}
}


In our product, we never show Events happening in the past - we base it 
completely on upcoming 'dates' - the same concept holds for Classes, but we 
base it on 'sessions.start'.

So 'session_data' and 'dates' are not separate documents at all, they're 
embedded in the 'activity' document. For Classes in particular, I'd like to 
pull out 'session_data' into another document type/mapping with a '_parent' 
to 'activity', but when we're returning results in our product we don't 
want to display duplicate items. In fact, we want to display the Class 
information along with the 'session_data' underneath in our search results 
listing 
(see 
http://www.chatterblock.com/camps-and-classes/victoria-british-columbia-c4098/?classes=y
 
for a working demo).

To get that result, I have to do some looping date checking in my Django 
view/template code to ensure I only display relevant (ie. upcoming, not 
past) Sessions.  This is the particularly inefficient bit I'm talking 
about. Sometimes there are MANY Sessions in an activity.

I have a feeling that the new inner_hits functionality might apply here, 
but I think I'll get into a 'grandparent' relationship scenario (two levels 
of parent relationships) as there's an existing relationship between 
'activity' and another document type called 'business' (who's running the 
Event or Class).

I really hope that this writeup wasn't too confusing! Everything 'works' at 
the moment, but Class listings are slower than other listings we've got, 
which is why I'm digging into inefficiencies to deal with this as we're 
growing nicely.

Thanks,
James

-- 
Please update your bookmarks! We moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/9a4e3285-5001-4bd0-a9cf-1178635ebd54%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Elastic Search: Fix cluster health Red using JDBC River

[James Crone]

Hello..

My Index Settings:

PUT usersdata
{
  "settings": {
"index": {
  "number_of_shards": 3,
  "number_of_replicas": 2,
  "analysis": {
"analyzer": {
  "analyzer_keyword": {
"tokenizer": "keyword",
"filter": "lowercase"
  }
}
  }
}
  },
  "mappings": {
"users": {
  "properties": {
"Name": {
  "analyzer": "analyzer_keyword",
  "type": "string"
}
  }
}
  }
}

Index River JDBC Settings:

POST _river/usersdatariver/_meta
{
  "type" : "jdbc",
   "jdbc" : {
 "url" : "jdbc:mysql://localhost:3306/alterduden",
  "user" : "root",
  "password" : "",
  "poll" : "6s",
  "index" : "usersdata",
  "type" : "users",
  "schedule":"0/10 * * ? * *",
  "strategy" : "simple",
  "sql" : "select ID,Name,DateTime,First_Name,Last_Name,ID as _id from 
users"
}
 }

I am using elasticsearch-1.5.0+marvel+elasticHQ plugins in LAN server but 
clustername and node name is different. When i create index its run fine 
for first time but when i restart elastic search status get RED.

{
  "cluster_name" : "mycluster",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 6,
  "active_shards" : 6,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 26,
  "number_of_pending_tasks" : 0
}

how to fix this problem?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/c8f33c6c-5e9e-4873-afcd-af9077825e64%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to fix cluster health yellow with Elastic Search

[James Crone]

Ok i understand. Thanks Pilato

On Thursday, April 30, 2015 at 3:38:47 PM UTC+5, David Pilato wrote:
>
> As I said, it’s not an issue. Your cluster is working well.
> But if you want to allocate one replica for each primary shard, you will 
> need one more node.
> If you set number of replica to 2, your cluster will be yellow again until 
> you start a new node.
>
> If you really want to change the node name, you can also set it using for 
> the first node
>
> bin/elasticsearch --node.name firstnode
>
> and this for the second
>
> bin/elasticsearch --node.name secondnode
>
> I said "If you really want to change the node name" because I guess that 
> you are doing tests on your local machine.
> So you basically don’t need to do that.
> If your plan is to do that in production, please don’t! Don’t run more 
> than one node per machine unless you have more than 64gb RAM.
>
>
>
>
> -- 
> *David Pilato* - Developer | Evangelist 
> *elastic.co <http://elastic.co>*
> @dadoonet <https://twitter.com/dadoonet> | @elasticsearchfr 
> <https://twitter.com/elasticsearchfr> | @scrutmydocs 
> <https://twitter.com/scrutmydocs>
>
>
>
>
>  
> Le 30 avr. 2015 à 11:22, James Crone > a 
> écrit :
>
> Thanks for your quick reply.
> Two nodes is necessary for this issue?
> And Can i define two nodes in single config file like:
> node:
>   name: firstnode
>   name: secondnode
> ? and just start by bin/elasticsearch
>
>
> On Thursday, April 30, 2015 at 2:12:22 PM UTC+5, David Pilato wrote:
>>
>> First this is not a problem. Just showing you that all primaries are 
>> allocated but nor replicas.
>>
>> How to solve it?
>>
>> Increase the number of nodes. Obviously elasticsearch doesn’t replicate 
>> on the same node
>> or
>> Reduce the number of replicas (you can do it live)
>> or
>> Live with it
>>
>> Now coming back to the fact that you are trying to start a new node, the 
>> output you printed shows that you have only one node running.
>>
>>  * "number_of_nodes" : 1,*
>>
>>
>> Note that to run another node, you are not forced to define a new 
>> elasticsearch.yml file.
>>
>> So running bin/elasticearch might be enough.
>>
>>
>> Not enough information to say more for now.
>>
>> -- 
>> *David Pilato* - Developer | Evangelist 
>> *elastic.co <http://elastic.co/>*
>> @dadoonet <https://twitter.com/dadoonet> | @elasticsearchfr 
>> <https://twitter.com/elasticsearchfr> | @scrutmydocs 
>> <https://twitter.com/scrutmydocs>
>>
>>
>>
>>
>>
>> Le 30 avr. 2015 à 10:50, James Crone  a écrit :
>>
>> Hello.!
>>
>>   Elastic search stuck in yellow status. I searched a lot about this 
>> problem and i find out same answer that to create two nodes. So i create 
>> two nodes by creating two config file elasticsearch.yml and 
>> elasticsearch-1.yml and start by command: 
>>
>> elasticsearch -Des.config=d:/elasticsearch-1.5.0/config/elasticsearch.yml
>> elasticsearch 
>> -Des.config=d:/elasticsearch-1.5.0/config/elasticsearch-1.yml
>>
>> Run this command Separately.
>>
>> first time its return green status. when i restarted again it shows again 
>> that:
>>
>> {
>>   "cluster_name" : "alterduden",
>>   "status" : "yellow",
>>   "timed_out" : false,
>>   "number_of_nodes" : 1,
>>   "number_of_data_nodes" : 1,
>>   "active_primary_shards" : 29,
>>   "active_shards" : 29,
>>   "relocating_shards" : 0,
>>   "initializing_shards" : 0,
>>   "unassigned_shards" : 26,
>>   "number_of_pending_tasks" : 0
>> }
>>
>> I am also trying to allocating shards but i failed this is necessary to 
>> create new index if i have to define shards? Or what should i do to fix 
>> this to *green* ?
>>
>>
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/cc7f3fcc-3dc8-44ec-a925-ccfc54fc5e03%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/elasticsearch/cc7f3fcc-3dc8-44ec-a925-ccfc54fc5e03%40googlegroups.com?utm_medium=email&a

Re: How to fix cluster health yellow with Elastic Search

[James Crone]

Thanks for your quick reply.
Two nodes is necessary for this issue?
And Can i define two nodes in single config file like:
node:
  name: firstnode
  name: secondnode
? and just start by bin/elasticsearch


On Thursday, April 30, 2015 at 2:12:22 PM UTC+5, David Pilato wrote:
>
> First this is not a problem. Just showing you that all primaries are 
> allocated but nor replicas.
>
> How to solve it?
>
> Increase the number of nodes. Obviously elasticsearch doesn’t replicate on 
> the same node
> or
> Reduce the number of replicas (you can do it live)
> or
> Live with it
>
> Now coming back to the fact that you are trying to start a new node, the 
> output you printed shows that you have only one node running.
>
>  * "number_of_nodes" : 1,*
>
>
> Note that to run another node, you are not forced to define a new 
> elasticsearch.yml file.
>
> So running bin/elasticearch might be enough.
>
>
> Not enough information to say more for now.
>
> -- 
> *David Pilato* - Developer | Evangelist 
> *elastic.co <http://elastic.co>*
> @dadoonet <https://twitter.com/dadoonet> | @elasticsearchfr 
> <https://twitter.com/elasticsearchfr> | @scrutmydocs 
> <https://twitter.com/scrutmydocs>
>
>
>
>
>  
> Le 30 avr. 2015 à 10:50, James Crone > a 
> écrit :
>
> Hello.!
>
>   Elastic search stuck in yellow status. I searched a lot about this 
> problem and i find out same answer that to create two nodes. So i create 
> two nodes by creating two config file elasticsearch.yml and 
> elasticsearch-1.yml and start by command: 
>
> elasticsearch -Des.config=d:/elasticsearch-1.5.0/config/elasticsearch.yml
> elasticsearch -Des.config=d:/elasticsearch-1.5.0/config/elasticsearch-1.yml
>
> Run this command Separately.
>
> first time its return green status. when i restarted again it shows again 
> that:
>
> {
>   "cluster_name" : "alterduden",
>   "status" : "yellow",
>   "timed_out" : false,
>   "number_of_nodes" : 1,
>   "number_of_data_nodes" : 1,
>   "active_primary_shards" : 29,
>   "active_shards" : 29,
>   "relocating_shards" : 0,
>   "initializing_shards" : 0,
>   "unassigned_shards" : 26,
>   "number_of_pending_tasks" : 0
> }
>
> I am also trying to allocating shards but i failed this is necessary to 
> create new index if i have to define shards? Or what should i do to fix 
> this to *green* ?
>
>
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearc...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/cc7f3fcc-3dc8-44ec-a925-ccfc54fc5e03%40googlegroups.com
>  
> <https://groups.google.com/d/msgid/elasticsearch/cc7f3fcc-3dc8-44ec-a925-ccfc54fc5e03%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/dbab7954-5e45-4904-a60d-bcb3c1377f49%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

How to fix cluster health yellow with Elastic Search

[James Crone]

Hello.!

  Elastic search stuck in yellow status. I searched a lot about this 
problem and i find out same answer that to create two nodes. So i create 
two nodes by creating two config file elasticsearch.yml and 
elasticsearch-1.yml and start by command: 

elasticsearch -Des.config=d:/elasticsearch-1.5.0/config/elasticsearch.yml
elasticsearch -Des.config=d:/elasticsearch-1.5.0/config/elasticsearch-1.yml

Run this command Separately.

first time its return green status. when i restarted again it shows again 
that:

{
  "cluster_name" : "alterduden",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 29,
  "active_shards" : 29,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 26,
  "number_of_pending_tasks" : 0
}

I am also trying to allocating shards but i failed this is necessary to 
create new index if i have to define shards? Or what should i do to fix 
this to *green* ?


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/cc7f3fcc-3dc8-44ec-a925-ccfc54fc5e03%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Sort Results by category and occurrences in other indexes

[James Radke]

Good morning,

I am new to ElasticSearch, and have the basics of querying, sorting, etc. 
working  with some test data.  

I have a concept that I would like to solve, but am not really sure how to 
do it, so I thought I would post here and see if you all can help me figure 
this out.

I have the following indexes (note, not all fields in the indexes are 
identified:


Categories Index (we have approximately 4 main category groupings, and 94 
different sub-category groupings)

- Id (int, identifies the category)
- Heading (character varying, heading for the category)
- ParentId (int, set to zero if a main category item, otherwise set to 
which item it is a subcategory item of)


Products Index (approximately 60,000 in test elasticsearch, with a 3 node 
cluster)

- Sku (character varying, identifies the product)
- Level1CatId  (int, identifies which main category the product belongs to)
- Level1CatHeading (character varying, main category heading)
- Level2CatId  (int, identfies which sub-category the product belongs to)
- Level2CatHeading (character varying, sub-category heading)


LogEvents Index

- DateTime (date, date and time the activity occurred)
- Type (character varying, can be either 'OfferClickThrough', 
'SearchResultClickThrough', 'AdClickThrough' amongst others like 'Login', 
'Logout', 'Errors')
- Sku (product identifier)
- SubCategoryId (sub categoryId the part belongs to)
   

What I would like to do is create results that look like:


Results {
 Category {
  Id,
  Heading,
  Products
   Product {
Sku
Description,
Price,
etc.
   }
   Product {
Sku,
Destination,
Price.
etc.
   }
   
  }
 }
 Category {
   ...
 }
 .
}


where, in English, it works like:

1.) Results are first split into categories.

2.) Within each category, products should first be sorted by sku based on 
the the most recent LogEvents Sku with types of 'OfferClickThrough', 
'SearchResultsClickThrough', and 'AdClickThrough').  After that, products 
should be randomized so the same items are not always presented the same 
every time the query has been run.

3.) Category groupings in the Results should first be sorted by the most 
recent LogEvents SubCategoryId with the types of 'OfferClickThrough', 
'SearchResultsClickThrough', and 'AdClickThrough' .  After that, the 
remaining categories should be randomized so they do not always come back 
in the same order every time the query has been run.


Is this doable with ElasticSearch?

Thanks,

Jim

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/16e004b2-be86-4a27-b43e-f00ad8d056be%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Adding a sycronisation key to elasticsearch.

[James]

Hi,
 
I have a system where I grab lots of XML files throughout the day. I keep a 
"sycned.xml" and a load of datestamped files too. Synced.xml is the latest 
file which has been sycned with elasticsearch.
 
I run a program which works out the difference between the latest 
timestamped file and "synced.xml" and depending on what has been added or 
removed in the latest timestamped file, I update elasticsearch. 
 
This means out of a large XML file, say 500,000 items, I only have to add 
the 10,000 or so that have changed. 
 
Can anyone recommend a way that I can check that elasticsearch and my 
script is syncronised? In my mind I was thinking of generating a hash key 
which I add to elasticsearch and log on my script. And if they don't match 
then I know they are out of sync. That said, how do I achieve this?
 
Thank you for any help.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/6ea631b5-f5f4-4ff2-9182-4251cd55e387%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Update River Settings MYSQL JDBC

[James Crone]

Thank You Jorg Prante and thank you for your quick reply.

On Friday, April 17, 2015 at 4:31:54 PM UTC+5, Jörg Prante wrote:
>
> You must  delete the river instance userentriessdatariver, and create a 
> new one.
>
> Jörg
>
> On Fri, Apr 17, 2015 at 12:51 PM, James Crone  > wrote:
>
>> Hi..
>>
>> I am new in elastic search and using 
>> https://github.com/jprante/elasticsearch-jdbc and my river setting is: 
>>
>> PUT _river/userentriessdatariver/_meta
>> {
>>   "type" : "jdbc",
>>"jdbc" : {
>>  "url" : "jdbc:mysql://localhost:3306/alterduden",
>>   "user" : "root",
>>   "password" : "",
>>   "poll" : "6s",
>>   "index" : "userentriesdata",
>>   "type" : "userentries",
>>   "schedule":"0/10 * * ? * *",
>>   "strategy" : "simple",
>>   "sql" : "select Name,DateTime,Categories_ID, ID as _id from 
>> userentries"
>> }
>>  } 
>>
>>
>> I just want to update sql command and schedule for my existing index. how 
>> can i update it? Plz help me.
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/ccc3db89-9879-40bd-b5c6-f5214269e3ed%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/elasticsearch/ccc3db89-9879-40bd-b5c6-f5214269e3ed%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/a5598e91-ddd4-45dd-989e-f56b31843921%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Update River Settings MYSQL JDBC

[James Crone]

Hi..

I am new in elastic search and using 
https://github.com/jprante/elasticsearch-jdbc and my river setting is: 

PUT _river/userentriessdatariver/_meta
{
  "type" : "jdbc",
   "jdbc" : {
 "url" : "jdbc:mysql://localhost:3306/alterduden",
  "user" : "root",
  "password" : "",
  "poll" : "6s",
  "index" : "userentriesdata",
  "type" : "userentries",
  "schedule":"0/10 * * ? * *",
  "strategy" : "simple",
  "sql" : "select Name,DateTime,Categories_ID, ID as _id from 
userentries"
}
 } 


I just want to update sql command and schedule for my existing index. how 
can i update it? Plz help me.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/ccc3db89-9879-40bd-b5c6-f5214269e3ed%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [logstash-users] Evaluating Moving to Discourse - Feedback Wanted

[James Green]

What's the missing is a culture in which consumers put questions directly
into documentation for answering by the next person who reads it and knows
the answer.

Note that it's not good enough to implement an append-only authorship for
Q&As, the question and the answer need to feed back into a re-wording of
the original documentation.

Using plain text sources for documentation such as asciidoc within a simple
editing suite (github is just about good enough) lowers the barriers.


On 16 April 2015 at 09:28, Simon Lundström  wrote:

> On Tue, 2015-04-14 at 09:15:25 +0100, James Green wrote:
> […]
> > It was never intended to document software use, which is what support
> lists
> > like this end up being used for. If I want to know how to do something or
> > what I might be doing wrong I'd prefer to place the question in precisely
> > the correct way and have it answered as part of the ongoing documentation
> > effort for others to read. This ongoing refinement of knowledge is what
> > ultimately drives forward adoption. A mailing list of far too fragmented
> to
> > achieve this - wrong tool for the job.
>
> I don't think forums (and thus Discourse) is the solution for this
> either. StackOverflow et.al. is OK but it's not really a good medium
> IMO. The project should take ownership of their docs and discussions so
> they can be correct and not just guessing.
>
> Official documentation which can be contributed to (via PRs is OK for
> me) by the community is a better solution IMO. This demands a clear
> structure from the project where things are supposed to be (i.e.
> "reference documentation for plugin X" vs. "this is how I solved X" vs.
> "What's best practice for LS/ES/plugin X") so we don't get yet another
> messy place where noone can find anything and it's an exhausting and
> daunting experience to find (or add) anything.
>
> > However many times have we asked Google a question and come back with a
> > mailing list post four-five years old which appears to be on the money,
> yet
> > the answer no longer appears to apply?
>
> I agree and even worse is when noone has answered it all and it's the
> only hit. Or my favourite: when your (generic and/or exact) search query
> yields no hits at all.
>
> BR,
> - Simon
>
> --
> Remember: if a new user has a bad time, it's a bug in logstash.
> ---
> You received this message because you are subscribed to the Google Groups
> "logstash-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to logstash-users+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAMH6%2Baz9e5duKcAMqCRuWy8ZhbhQW6q4QzNEdZXutrqUy22sJQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Querystring search: Tokens are out of order

[James Macdonald]

You analysis of what is going on sounds correct. However, Elasticsearch's
results are also correct. When it analyzes the search string, your query
becomes a match query on "foo" AND "bar", which matches any document
containing both of those terms. Most queries against analyzed fields do not
respect the original ordering of the terms.

One thing you could try is looking into the match_phrase query (
http://www.elastic.co/guide/en/elasticsearch/guide/master/phrase-matching.html)
which is aware of the ordering of the terms. Using the base match_phrase
query for "foo bar" will not match either "foo xyz bar" or "bar xyz foo".
If you still need to match things like  "foo xyz bar" you may be able to do
that using the slop parameter, depending on what exactly the use case is.

James

On Tue, Apr 14, 2015 at 2:03 PM, Dave Reed  wrote:

> I have the following search:
>
> {
>   "query": {
> "filtered": {
>   "query": {
> "query_string": {
>   "default_operator": "AND",
>   "query": "details:foo\\-bar"
> }
>   },
>   "filter": {
> "term": {
>   "deleted": false
> }
>   }
> }
>   }
> }
>
>
>
> The details field is analyzed using pattern tokenizer, as so:
>
> settings: {
>   index.analysis.analyzer.letterordigit.pattern: "[^\\p{L}\\p{N}]+",
>   index.analysis.analyzer.letterordigit.type: "pattern"
> }
>
>
> This breaks the field into tokens separated by any non-letter or
> non-numeric character.
>
> But the user is searching for "foo-bar" which contains a non alphanumeric
> character. I assume, but correct me if I'm wrong, that ES will apply the
> same analyzer to that string. So it is broken into two tokens: ["foo",
> "bar"], and then the default_operator kicks in and essentially turns the
> query into "details:foo AND detail:bar".
>
> My problem is that it will match documents containing "foo xyz bar" and
> "bar xyz foo" -- in the latter case, the tokens are in the reverse order
> from the user's search. I'm fine with it matching the former, but it's a
> stretch to convince the user that the latter is intended.
>
> The search string is provided by the user, so I can't really build a
> complex query with different query types, hence the basic querystring
> search.
>
> Any advice or corrections to my assumptions is appreciated!
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/4a204214-f209-48dd-a13a-96463609ad7d%40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/4a204214-f209-48dd-a13a-96463609ad7d%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAABsnTZWNp65WzwYsZVZz%3DiHon7WW90EO8SUKbnB4aHuKcd-og%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Aggregation not limited to filter?

[James Green]

Indeed. I had used postFilter to add my filters. The documentation for
filters doesn't show how to use a query with a matchAll and a bunch of
filters so I blindly followed IDE auto-complete.

Lesson learned.

On 10 April 2015 at 21:17, James Macdonald 
wrote:

> I had a similar problem recently and solved it by moving my filter into a
> filtered query (leaving the query as a match_all), see documentation here
> http://www.elastic.co/guide/en/elasticsearch/reference/1.5/query-dsl-filtered-query.html
> .
>
> I am not certain why filters do not restrict the scope of the aggregates,
> but queries do, but I suspect it interprets the filter (not wrapped in a
> filtered_query) as a post_filter (
> http://www.elastic.co/guide/en/elasticsearch/reference/1.x/search-request-post-filter.html).
> Maybe someone else actually knows why.
>
>
> Hope that helps,
> James
>
> On Fri, Apr 10, 2015 at 11:39 AM, James Green 
> wrote:
>
>> I must be doing something stupid!
>>
>> Using the Java client I can perform a search with a filter and iterate
>> over the hits. I see exactly the right source documents.
>>
>> If I add an aggregation, I see the expected keyAsText string but the
>> docCount reflects the volume if the filter had not been applied.
>>
>> I expected the aggregation to be restricted to the results within that
>> filter?
>>
>> Thanks,
>>
>> James
>>
>>  --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CAMH6%2BaxkmZVfDhkJW-bWPrRs5BMzTem-2zCQRWeF%2BLQCR2L5sA%40mail.gmail.com
>> <https://groups.google.com/d/msgid/elasticsearch/CAMH6%2BaxkmZVfDhkJW-bWPrRs5BMzTem-2zCQRWeF%2BLQCR2L5sA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAABsnTbD0JgcpMCMWuzjVC1W3C-pt6pC6PJG0xT31O44MZQs%3DA%40mail.gmail.com
> <https://groups.google.com/d/msgid/elasticsearch/CAABsnTbD0JgcpMCMWuzjVC1W3C-pt6pC6PJG0xT31O44MZQs%3DA%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAMH6%2BaxhjdOQrfqy9Upsvnh%2B%2BgHmFDwqw%3DQqjHW2Z7DUm7BvJg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Aggregation not limited to filter?

[James Green]

Indeed. I had used postFilter to add my filters. The documentation for
filters doesn't show how to use a query with a matchAll and a bunch of
filters so I blindly followed IDE auto-complete.

Lesson learned.

On 10 April 2015 at 21:17, James Macdonald 
wrote:

> I had a similar problem recently and solved it by moving my filter into a
> filtered query (leaving the query as a match_all), see documentation here
> http://www.elastic.co/guide/en/elasticsearch/reference/1.5/query-dsl-filtered-query.html
> .
>
> I am not certain why filters do not restrict the scope of the aggregates,
> but queries do, but I suspect it interprets the filter (not wrapped in a
> filtered_query) as a post_filter (
> http://www.elastic.co/guide/en/elasticsearch/reference/1.x/search-request-post-filter.html).
> Maybe someone else actually knows why.
>
>
> Hope that helps,
> James
>
> On Fri, Apr 10, 2015 at 11:39 AM, James Green 
> wrote:
>
>> I must be doing something stupid!
>>
>> Using the Java client I can perform a search with a filter and iterate
>> over the hits. I see exactly the right source documents.
>>
>> If I add an aggregation, I see the expected keyAsText string but the
>> docCount reflects the volume if the filter had not been applied.
>>
>> I expected the aggregation to be restricted to the results within that
>> filter?
>>
>> Thanks,
>>
>> James
>>
>>  --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CAMH6%2BaxkmZVfDhkJW-bWPrRs5BMzTem-2zCQRWeF%2BLQCR2L5sA%40mail.gmail.com
>> <https://groups.google.com/d/msgid/elasticsearch/CAMH6%2BaxkmZVfDhkJW-bWPrRs5BMzTem-2zCQRWeF%2BLQCR2L5sA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAABsnTbD0JgcpMCMWuzjVC1W3C-pt6pC6PJG0xT31O44MZQs%3DA%40mail.gmail.com
> <https://groups.google.com/d/msgid/elasticsearch/CAABsnTbD0JgcpMCMWuzjVC1W3C-pt6pC6PJG0xT31O44MZQs%3DA%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAMH6%2BaxDfHvCicw5rewNOAun5Vy2qZe8X_awGD3wR8B-vVZY-A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Aggregation not limited to filter?

[James Macdonald]

I had a similar problem recently and solved it by moving my filter into a
filtered query (leaving the query as a match_all), see documentation here
http://www.elastic.co/guide/en/elasticsearch/reference/1.5/query-dsl-filtered-query.html
.

I am not certain why filters do not restrict the scope of the aggregates,
but queries do, but I suspect it interprets the filter (not wrapped in a
filtered_query) as a post_filter (
http://www.elastic.co/guide/en/elasticsearch/reference/1.x/search-request-post-filter.html).
Maybe someone else actually knows why.


Hope that helps,
James

On Fri, Apr 10, 2015 at 11:39 AM, James Green 
wrote:

> I must be doing something stupid!
>
> Using the Java client I can perform a search with a filter and iterate
> over the hits. I see exactly the right source documents.
>
> If I add an aggregation, I see the expected keyAsText string but the
> docCount reflects the volume if the filter had not been applied.
>
> I expected the aggregation to be restricted to the results within that
> filter?
>
> Thanks,
>
> James
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAMH6%2BaxkmZVfDhkJW-bWPrRs5BMzTem-2zCQRWeF%2BLQCR2L5sA%40mail.gmail.com
> <https://groups.google.com/d/msgid/elasticsearch/CAMH6%2BaxkmZVfDhkJW-bWPrRs5BMzTem-2zCQRWeF%2BLQCR2L5sA%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAABsnTbD0JgcpMCMWuzjVC1W3C-pt6pC6PJG0xT31O44MZQs%3DA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Incorrect Date Range Query Result

[James Crone]

I have try but same problem. I think this add +1 in date because when i try 
date b/w 2015-03-01 to 2015-03-08 then it return null. and when i try 
2015-03-10 it returns data of 2015-03-10 and 2015-03-11 too. What You say?

On Thursday, April 9, 2015 at 7:44:51 PM UTC+5, Jan Wolf wrote:
>
> Hmmm strange. Well I if you don't need scoring you should also use filters 
> instead of queries. Try this one (works for me):
>
> "query" : {
>   "filtered" : {
> "query" : {
>   "query_string" : {
> "query" : "*"
>   }
> },
> "filter" : {
>   "bool" : {
> "must" : {
>   "range" :  {
> "DateTime" : {
>   "gt" : "",
>   "lte" : ""
> }
>   }
> }
>   }
> }
>   }
> }
>
> Am Donnerstag, 9. April 2015 15:27:08 UTC+2 schrieb James Crone:
>>
>> Same problem. :(
>>
>> On Thursday, April 9, 2015 at 6:25:25 PM UTC+5, Jan Wolf wrote:
>>>
>>> I would try "must" instead of "should" :)
>>>
>>> Am Dienstag, 7. April 2015 10:55:01 UTC+2 schrieb James Crone:
>>>>
>>>> Hello!
>>>>
>>>> My query is this:
>>>>
>>>> GET _search
>>>> {
>>>>   "query": {
>>>> "bool": {
>>>>   "should": [
>>>>{
>>>>   "range": {
>>>> "DateTime": {
>>>>   "gt": "2015-03-01",
>>>>   "lte": "2015-03-09"
>>>> }
>>>>   }
>>>> }
>>>>   ]
>>>> }
>>>>   }
>>>> }
>>>> Result is:
>>>>
>>>> {
>>>>"took": 2,
>>>>"timed_out": false,
>>>>"_shards": {
>>>>   "total": 11,
>>>>   "successful": 11,
>>>>   "failed": 0
>>>>},
>>>>"hits": {
>>>>   "total": 1,
>>>>   "max_score": 1,
>>>>   "hits": [
>>>>  {
>>>> "_index": "completedata",
>>>> "_type": "users",
>>>> "_id": "1",
>>>> "_score": 1,
>>>> "_source": {
>>>>"ID": 1,
>>>>"User_ID": 1,
>>>>"Name": "Saudia Arabia Bomb",
>>>>"Text": "Saudia Arabia Bomb",
>>>>"Status": "online",
>>>>"DateTime": "2015-03-10T00:00:00.000+05:00",
>>>>"categories_ID": 1
>>>> }
>>>>  }
>>>>   ]
>>>>}
>>>> }
>>>>
>>>> mapping for date is:
>>>>
>>>> "DateTime" : {
>>>> "type" : "date",
>>>> "format" : "dateOptionalTime"
>>>>   }
>>>>
>>>> It returns me the result of "2015-03-10".How can i get exact result by 
>>>> matching Date? Any Idea?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e9b1cf3f-d864-4142-935c-c35c9cf9a50b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Incorrect Date Range Query Result

[James Crone]

Same problem. :(

On Thursday, April 9, 2015 at 6:25:25 PM UTC+5, Jan Wolf wrote:
>
> I would try "must" instead of "should" :)
>
> Am Dienstag, 7. April 2015 10:55:01 UTC+2 schrieb James Crone:
>>
>> Hello!
>>
>> My query is this:
>>
>> GET _search
>> {
>>   "query": {
>> "bool": {
>>   "should": [
>>{
>>   "range": {
>> "DateTime": {
>>   "gt": "2015-03-01",
>>   "lte": "2015-03-09"
>> }
>>   }
>> }
>>   ]
>> }
>>   }
>> }
>> Result is:
>>
>> {
>>"took": 2,
>>"timed_out": false,
>>"_shards": {
>>   "total": 11,
>>   "successful": 11,
>>   "failed": 0
>>},
>>"hits": {
>>   "total": 1,
>>   "max_score": 1,
>>   "hits": [
>>  {
>> "_index": "completedata",
>> "_type": "users",
>> "_id": "1",
>> "_score": 1,
>> "_source": {
>>"ID": 1,
>>"User_ID": 1,
>>"Name": "Saudia Arabia Bomb",
>>"Text": "Saudia Arabia Bomb",
>>"Status": "online",
>>"DateTime": "2015-03-10T00:00:00.000+05:00",
>>"categories_ID": 1
>> }
>>  }
>>   ]
>>}
>> }
>>
>> mapping for date is:
>>
>> "DateTime" : {
>> "type" : "date",
>> "format" : "dateOptionalTime"
>>   }
>>
>> It returns me the result of "2015-03-10".How can i get exact result by 
>> matching Date? Any Idea?
>>
>>
>>
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/65a49d2f-8aa8-4252-84ca-5bb6391bf6b3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Kibana: Mark warnings as "solved"

[James Green]

Couldn't you update the document with a flag on a field?

On 8 April 2015 at 09:43, Thomas Güttler  wrote:

> We are evaluating if ELK is the right tool for our logs and event messages.
>
> We need a way to mark warnings as "done". All warnings of this type should
> be invisible in the future.
>
> Use case:
>
> There was a bug in our code and the dev team has created a fix. Continuous
> Integration is running,
> and soon the bug in the production system will be gone.
>
> We need a way to mark the warnings as "this type of warning is already
> handled, and the
> fix will be in the production system during the next three hours".
>
> Can you understand what I want?
>
> How to handle this with ELK?
>
> Just removing these logs from ElasticSearch is not a solution, since
> during the next hours (after
> setting the flag "done") new events can still come into the system.
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/ff5e0583-3f1d-4ba4-af38-ee0a4823afc2%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAMH6%2BayGJGhn0VfBmJaeLEiaXyXjG%2BU-spHdhia4Rfu%3DUjTfcw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: river jdbc plugin install for windows-not working

[James Green]

As discussed elsewhere please avoid Rivers as they are deprecated for
removal.

On 6 April 2015 at 13:36, Sanu Vimal  wrote:

> Hi Jorg,
>
> I have found what was the issue. I was just trying the POC on MySQL but
> the real requirement is in oracle11g. I have installed through plugin.bat
> after setting up ES_HOME in my system variable.
>
> Thanks
> Sanu
>
> On Monday, April 6, 2015 at 9:06:59 AM UTC+5:30, Sanu Vimal wrote:
>
>> Hi
>>  Everything worked on a linux machine. Now i need to use the river jdbc
>> plugin in the windows but there is not much documentation or support. I
>> tried everything created the plugin folder with river jdbc in the jdbc
>> folder added all type of mysql connectors in the folder i also added the
>> jar file in the lib of elastic search. I also added the jar files in the
>> class path of env variables. Still i get
>>
>> [2015-04-05 22:49:12,568][ERROR][river.jdbc.SimpleRiverSource] while
>> opening
>> read connection: jdbc:mysql://localhost:3306/test No suitable driver
>> found for jdbc
>> :mysql://localhost:3306/test
>> java.sql.SQLException: No suitable driver found for
>> jdbc:mysql://localhost:3306/
>> test
>>
>> I am using cygwin to eecute my river
>>
>> curl -XPUT 'localhost:9200/_river/my_jdbc_river/_meta' -d '{
>> "type" : "jdbc",
>> "jdbc" : {
>> "url" : "jdbc:mysql://localhost:3306/test",
>> "user" : "root",
>> "password" : "",
>> "sql" : "select * from test.orders"
>> }
>> }'
>>
>> In mysql the connection name is "Sanu" and the data base name is test.
>> Please help or reffer anything.
>>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/939abc32-1a4d-4efd-a324-c8b32a7e1fec%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAMH6%2BawFii%2BjkMramzHRPhAdr2GdqMQH%3DwH_hVO-qLqhWt%2BKjg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] Deprecating Rivers

[James Green]

Also note that the current documentation lists a load of river plugins
without stating that they are officially deprecated as an idea:

http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-plugins.html

Rather concerned that people are still adopting rivers due to the apparent
continued promotion of them. If you have not already, I suggest each time a
node fires up and detects a river plugin it issues at least a warning in a
log to remove it in favour of alternatives such as logstash. At least the
official rivers are marked deprecated in their readme but not everyone will
spot this.


On 1 April 2015 at 23:30, Mark Walkom  wrote:

> Hi All,
> We've recently announced that Rivers in Elasticsearch are actively
> deprecated as of the 1.5 release, the 2.0 release will still maintain that
> functionality to ease migration, but we recommend you start to leverage the
> official clients to replace any Rivers you may have in use.
>
> If you have feedback around this process then feel free to add a comment
> to this Github issue -
> https://github.com/elastic/elasticsearch/issues/10345
>
> And for more information please see this blog post on our site -
> https://www.elastic.co/blog/deprecating_rivers
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAEYi1X90gyaxCnV9ygkdEAJirv_4p8F3E_1J9jQU_axn9MBCGw%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAMH6%2Bazr1yamseHfSD9HriTqyos6uLJ7zX3qa%2B7oqom24qCeCA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Incorrect Date Range Query Result

[James Crone]

Hello!

My query is this:

GET _search
{
  "query": {
"bool": {
  "should": [
   {
  "range": {
"DateTime": {
  "gt": "2015-03-01",
  "lte": "2015-03-09"
}
  }
}
  ]
}
  }
}
Result is:

{
   "took": 2,
   "timed_out": false,
   "_shards": {
  "total": 11,
  "successful": 11,
  "failed": 0
   },
   "hits": {
  "total": 1,
  "max_score": 1,
  "hits": [
 {
"_index": "completedata",
"_type": "users",
"_id": "1",
"_score": 1,
"_source": {
   "ID": 1,
   "User_ID": 1,
   "Name": "Saudia Arabia Bomb",
   "Text": "Saudia Arabia Bomb",
   "Status": "online",
   "DateTime": "2015-03-10T00:00:00.000+05:00",
   "categories_ID": 1
}
 }
  ]
   }
}

mapping for date is:

"DateTime" : {
"type" : "date",
"format" : "dateOptionalTime"
  }

It returns me the result of "2015-03-10".How can i get exact result by matching 
Date? Any Idea?





-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/3865ceef-bdc4-4f31-a1ff-e4c574497853%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Prefix Query Result Problem.

[James Crone]

prefix query is not returning the correct results..
in ISO Country data.. my query was

{"query":{"bool":{"must":[{"prefix":{"country.name
":"in"}}],"must_not":[],"should":[]}},"from":0,"size":50,"sort":[],"facets":{}}
and the returned result is:
{ took: 3 timed_out: false _shards: { total: 5 successful: 5 failed: 0 } 
hits: { total: 3 max_score: 1 hits: [ { _index: restdb _type: country _id: 
sX3-aNGBQaOjo-L7zNj7qQ _score: 1_source: { id: 519fcbce820ea8950373612b code
: ID name: INDONESIA } }{ _index: restdb _type: country _id: 
YMXpIxaPRzmkMZqaKQ36pw _score: 1_source: { id: 519fcbce820ea895037360e4 code
: IO name: *BRITISH INDIAN OCEAN TERRITORY* }}{ _index: restdb _type: 
country _id: Ok9MHCr4RLqjH6oFLX92EA _score: 1_source: { id: 
519fcbce820ea8950373612a code: IN name: INDIA }}
] } }

As, you can see, red coloured data, is not matching prefix query..
Am i correct i need to do something else.., if not please help me to find 
out correct way to do so, else, give me go ahead, to post this as bug.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/13ab0a36-4e87-4cc2-89e2-0abe39f13668%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Getting XML into ES efficiently

[James]

Hi,

Currently I am using scrapy to parse an XML file from an ftp server into 
elasticsearch. It works but seems quite a heavy weight solution and it uses a 
lot of memory too.

I am wondering if I am better off writing a plugin for ES instead. 

I have some questions:

A) It seems writing it in Python (since I'm a python guy) as a push plugin 
rather than a pull river makes sense, unless anyone has a reason why pull is 
better?

B) For simple importing (and slight modification such as trimming, language 
check etc) is it likely that an ES plugin is likely going to be a better 
solution to importing fairly large XML files or should I just leave scrapy to 
do it as it is doing at the moment?

Any help and advice would be appreciated as I start on this journey.

James

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/610e7f9b-3d23-44a9-b8f3-07deb262dd54%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] Deprecating Rivers

[James Green]

Good luck with this when "Elasticsearch Team" is continuing to email the
list with river plugin updates and asking for feature requests to their
github projects.


On 1 April 2015 at 23:30, Mark Walkom  wrote:

> Hi All,
> We've recently announced that Rivers in Elasticsearch are actively
> deprecated as of the 1.5 release, the 2.0 release will still maintain that
> functionality to ease migration, but we recommend you start to leverage the
> official clients to replace any Rivers you may have in use.
>
> If you have feedback around this process then feel free to add a comment
> to this Github issue -
> https://github.com/elastic/elasticsearch/issues/10345
>
> And for more information please see this blog post on our site -
> https://www.elastic.co/blog/deprecating_rivers
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAEYi1X90gyaxCnV9ygkdEAJirv_4p8F3E_1J9jQU_axn9MBCGw%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAMH6%2BaxMe0yPQyRCWB0e8RytsvPt67OFxoH71RBE%3Dh_GcEfA-Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

How to Backup Cluster by Time Scheduled:

[James Crone]

I have create a backup directory using FS Technique.

 

 PUT _snapshot/databackup {
   "type": "fs", 
   "settings": {
  "location": "/mount/backups/my_backup" 
   }}

Then:

 PUT _snapshot/my_backup/snapshot_2 {
   "indices": "completedata" }


Its working.But when my database table update or insert new records backup not 
update automatically.How Can i Do this auto
backup after 1 day or by defining time. 

Thanks in advance.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/3aa4d6ea-a0c5-427c-973d-90ea292700c8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ElasticSearch date format

[James Green]

http://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html

The T is a standard symbol, not exclusive to any one software project.
ElasticSearch uses Joda to parse dates, and it looks like the default
accepts ISO 8601 which I would expect.

On 31 March 2015 at 15:32,  wrote:

> HI James,
>
>   Thanks for quick reply..
>
>so my understanding is while inserting in to elasticsearch we need to
> use T as separator other wise it won't insert in to date field from java
> API because it expecting standard format of ISO 8601 am I right?
>
> Thanks
> phani
>
> On Tuesday, March 31, 2015 at 7:48:42 PM UTC+5:30, James Green wrote:
>>
>> I am speculating here that you should use the ISO 8601 format for dates
>> since this is quite common and has a timezone which yours lacks
>> (introducing ambiguity).
>>
>> The T is a common separator between the date and time parts.
>>
>> On 31 March 2015 at 14:26,  wrote:
>>
>>> Hi All,
>>>
>>>when i insert date to elastic search in the format of -MM-DD
>>> hh:mm:ss from java api getting date format exception.but when i use
>>> -MM-DDThh:mm:ss between them it is inserted fine.please explain me is T
>>> here elasticsearch standard format?
>>>
>>>   please explain me functionality of T in date.i used the default
>>> mapping of elasticsearch i didn't customized any thing.
>>>
>>>  mapping:
>>>
>>> "created": {
>>>   "type": "date",
>>>   "format": "dateOptionalTime"
>>>},
>>>
>>>
>>> Thanks
>>> phani
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/elasticsearch/94b29fca-637d-4d73-84fe-6af4b6300611%
>>> 40googlegroups.com
>>> <https://groups.google.com/d/msgid/elasticsearch/94b29fca-637d-4d73-84fe-6af4b6300611%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/0a7fd7e8-b244-42e9-8427-752474250b07%40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/0a7fd7e8-b244-42e9-8427-752474250b07%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAMH6%2BawqvJ_DOZUR2u7C%3DU7AvFFr%3DhyBoBTMRwCbdow7kZxOsg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: ElasticSearch date format

[James Green]

I am speculating here that you should use the ISO 8601 format for dates
since this is quite common and has a timezone which yours lacks
(introducing ambiguity).

The T is a common separator between the date and time parts.

On 31 March 2015 at 14:26,  wrote:

> Hi All,
>
>when i insert date to elastic search in the format of -MM-DD
> hh:mm:ss from java api getting date format exception.but when i use
> -MM-DDThh:mm:ss between them it is inserted fine.please explain me is T
> here elasticsearch standard format?
>
>   please explain me functionality of T in date.i used the default mapping
> of elasticsearch i didn't customized any thing.
>
>  mapping:
>
> "created": {
>   "type": "date",
>   "format": "dateOptionalTime"
>},
>
>
> Thanks
> phani
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/94b29fca-637d-4d73-84fe-6af4b6300611%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAMH6%2BawJXDAs175DFWH5T5z5%2B-UvJgppKjvqPRBehzMX2R%2Bz7A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch - Not require an exact match

[James]

Thank you for the reply. I thought it was much about making my search query
not require an exact match, rather than splitting down the words I am
searching against?

On 19 March 2015 at 12:30, Itamar Syn-Hershko  wrote:

> You should use this then:
> http://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-word-delimiter-tokenfilter.html
>
> --
>
> Itamar Syn-Hershko
> http://code972.com | @synhershko <https://twitter.com/synhershko>
> Freelance Developer & Consultant
> Lucene.NET committer and PMC member
>
> On Thu, Mar 19, 2015 at 2:26 PM, James  wrote:
>
>> Currently I have an item in my elasticsearch index with the title:
>> *testing123*
>>
>> When I search for it, I can only get it returned if I search *testing123*
>> exactly. However, I want to be able to search *testing* and have it
>> returned too.
>>
>> How can I have it so the search must start with that term but also not be
>> an exact match?
>>
>> Any help would be appreciated.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/de42c52d-1566-461e-b578-594aa963a498%40googlegroups.com
>> <https://groups.google.com/d/msgid/elasticsearch/de42c52d-1566-461e-b578-594aa963a498%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/psZ8iAVOziM/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZvcvCYD0_SRY_QcHNcytaj7KcmCH2itJcUsPCTc2FjsrQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZvcvCYD0_SRY_QcHNcytaj7KcmCH2itJcUsPCTc2FjsrQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAPng%3D3f7pQOm4HPkxtec3or%2Ba4ZUVpC1USb7tWMaD9cvicQ8gQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Elasticsearch - Not require an exact match

[James]

Currently I have an item in my elasticsearch index with the title: 
*testing123*

When I search for it, I can only get it returned if I search *testing123* 
exactly. However, I want to be able to search *testing* and have it 
returned too.

How can I have it so the search must start with that term but also not be 
an exact match?

Any help would be appreciated.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/de42c52d-1566-461e-b578-594aa963a498%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Truncate response

[James]

Hi,

I am just wondering if there is an efficient and straight forward way to 
tell elasticsearch to truncate the reponse it sends. For example one of my 
fields is a "description". Is it possible to have it to when it sends the 
search response it only sends the first 200 charts of the description (and 
yet still keeps the full description in the index to search against).

Any help would be appreciated as searches on the subject turned up nothing.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/24d86390-5810-4e9e-b68a-efbe6c8b3751%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Leaving out information from the response

[James]

Thank you

On 25 February 2015 at 08:17, Thomas Matthijs  wrote:

>
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-fields.html
>
> On Wed, Feb 25, 2015 at 9:13 AM, James  wrote:
>
>> Hi,
>>
>> I want to have certain data in my elasticsearch index but I don't want it
>> to be returned with a query. At the moment it seems to return every bit of
>> data I have for each index and then I use my PHP application to hide it. Is
>> it possible to select what fields elasticsearch returns in it's response to
>> my PHP application.
>>
>> For example for each time:
>>
>> Name
>> Location
>> Description
>> Keywords
>> Unique ID
>> Create date
>>
>> I just want to have in the response from elasticsearch:
>>
>> Name
>> Location
>> Description
>>
>>
>>  --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/089c53f5-0aa4-48b5-acb4-df4d6ccfee13%40googlegroups.com
>> <https://groups.google.com/d/msgid/elasticsearch/089c53f5-0aa4-48b5-acb4-df4d6ccfee13%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/EurLxszzenc/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CABY_-Z4x%3DgOq1EbtvsLLRgQn1Ad8Zd5QzpubZaL5KA98p7J2Xw%40mail.gmail.com
> <https://groups.google.com/d/msgid/elasticsearch/CABY_-Z4x%3DgOq1EbtvsLLRgQn1Ad8Zd5QzpubZaL5KA98p7J2Xw%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAPng%3D3eNd3Gw8OdJpWPBF4tvSwWAKcuYasqCVmi7vdgYvLf0ow%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Leaving out information from the response

[James]

Hi,

I want to have certain data in my elasticsearch index but I don't want it 
to be returned with a query. At the moment it seems to return every bit of 
data I have for each index and then I use my PHP application to hide it. Is 
it possible to select what fields elasticsearch returns in it's response to 
my PHP application.

For example for each time:

Name
Location
Description
Keywords
Unique ID
Create date

I just want to have in the response from elasticsearch:

Name
Location
Description


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/089c53f5-0aa4-48b5-acb4-df4d6ccfee13%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch Sorting before applying filters?

[James Ma]

Hey so we figured out that the constant slow query speed when sorting is
because we were clearing the field data before every query which made ES
have to load the field data back to cache each time.

Note ES doesn't just load the field data for the hits of the specific query
when sorting, it will load the field data for every doc in the entire
index, which was causing our performance issue.

On Tue, Feb 17, 2015 at 10:32 AM, Bipul Dutta 
wrote:

> Wondering if adding search_type=dfs_query_then_fetch will change anything.
>
> On Tue, Feb 17, 2015 at 9:59 AM, Jay  wrote:
>
>> We are working in PyES, our queries currently look like this, but we have
>> tried putting the sort directly into the body too. It made no difference
>>
>> def termQuery(org, field, term, write, size=500):
>> q = {
>> "query": {
>> "filtered": {
>> "filter": {
>> "and": [
>> {"term": {"Org ID": org}},
>> {"term": {field: term}}
>> ]
>> }
>> }
>> }
>> }
>> r = es.search(index='_all', body=q, size=size, sort='Timestamp:desc')
>>
>>
>>
>> On Monday, February 16, 2015 at 10:57:36 PM UTC-8, Mark Walkom wrote:
>>>
>>> It probably depends on what your query looks like, it'd help if you
>>> could gist/pastebin/etc a link to a sample.
>>>
>>> On 17 February 2015 at 11:04,  wrote:
>>>
 Hello,

 I have noticed that when I'm applying sorting in my queries, the
 return times are remaining relatively constant at ~1.5s no matter what
 filters I apply (Even if I get 0 hits). It seems like ES might be
 sorting the entire index, then applying the filters after.

 As a test I checked how long it took ES to sort the entire index and
 it was about 2s.

 Here are some results for different queries comparing how many hits
 they got to how much time they took(ms):

 Took: 1563
 Hits: 7000

 Took: 1660
 Hits: 239360

 Took: 1657
 Hits: 0

 As you can see the time the queries are taking remains relatively
 constant when I have sorting on no matter how many hits I get. What is
 happening here?

 As a baseline if I remove sorting I get the following:

 Took: 150
 Hits: 7000

 Took: 134
 Hits: 239360

 Took: 0
 Hits: 0

 *All these tests were done with a return size of 500, so I would expect
 queries with more hits to return slightly faster

 --
 You received this message because you are subscribed to the Google
 Groups "elasticsearch" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to elasticsearc...@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/
 msgid/elasticsearch/e0b97088-6454-4fe2-8707-c0f629aecf0a%
 40googlegroups.com
 
 .
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>  --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/f96a81dc-0551-452e-80ef-9e128e08d8e0%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> This message contains confidential information and is intended only for
> the individual named. If you are not the named addressee, any delivery,
> disclosure, dissemination or distribution of this e-mail is unlawful and
> strictly prohibited.   Please notify the sender immediately by e-mail if
> you have received this e-mail by mistake and delete this e-mail from your
> system. E-mail transmissions cannot be guaranteed to be secure or without
> error, as information could be intercepted, corrupted, lost, destroyed,
> arrive late, be incomplete, or contain viruses. The sender therefore does
> not accept liability for any errors or omissions in the contents of this
> message, which errors or omissions arise as a result of e-mail
> transmission. If verification is required, please request a hard-copy
> version. (Res.Sys-dct1)
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/xPewWFV-LKo/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view 

Transport Error 400 - Failed to derive xcontent

[James]

Hi all,

I'm trying to run a scrapy crawler to feed elasticsearch but I seem to be 
getting this error and I can't understand how to begin debugging this. 

File 
"/var/www/app/venv/local/lib/python2.7/sitepackages/elasticsearch/connection/base.py",
 
line 97, in _raise_error HTTP_EXCEPTIONS.get(status_code, 
TransportError)(status_code, error_message, additional_info) 
elasticsearch.exceptions.RequestError: TransportError(400, u'Failed to 
derive xcontent from 
org.elasticsearch.common.bytes.ChannelBufferBytesReference@a')

Can anyone help me on my way to understand what is going wrong and how I 
might start to debug it? 

Any help would be appreciated.

James


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1bc780e4-6930-49e4-a24b-6f01a20d51c4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Integration test failing after upgrade to 1.4

[James Ekstrom]

We had a similar issue with a plugin regarding this. I found the solution 
here on this post from 
2012: 
http://elasticsearch-users.115913.n3.nabble.com/Running-ES-Client-inside-ES-Plugin-td3925864.html

Injecting the client in setClient was throwing the AllocationService 
circular dependency exception. Changing it to retrieve from injector solved 
the issue.

On Monday, November 17, 2014 at 8:52:34 PM UTC-6, Deepak wrote:
>
> Hi,
>
> After upgrading to ES 1.4, integration tests for our ES plugin started 
> failing. The plugin works fine when deployed directly to ES. The issue 
> seems to be only with integration tests which extend 
> ElasticsearchIntegrationTest class. 
>
> I have uploaded sample project to recreate the issue @ 
> https://github.com/dr2014/ES1.4_IntegrationTest_Issue
>
> There seems to be some problem with how Guice injects Client when 
> integration tests are executed.
>
> Any ideas as to how to resolve this issue?
>
> Thanks,
> Deepak
>
> Partial stacktrace:
>
> org.elasticsearch.common.inject.CreationException: Guice creation errors:
>
> 1) Tried proxying org.elasticsearch.discovery.DiscoveryService to support 
> a circular dependency, but it is not an interface.
>   while locating org.elasticsearch.discovery.DiscoveryService
> for parameter 7 at 
> org.elasticsearch.discovery.local.LocalDiscovery.(Unknown Source)
>   while locating org.elasticsearch.discovery.local.LocalDiscovery
>   while locating org.elasticsearch.discovery.Discovery
> for parameter 2 at 
> org.elasticsearch.discovery.DiscoveryService.(Unknown Source)
>   while locating org.elasticsearch.discovery.DiscoveryService
> for parameter 1 at 
> org.elasticsearch.cluster.service.InternalClusterService.(Unknown 
> Source)
>   while locating org.elasticsearch.cluster.service.InternalClusterService
>   while locating org.elasticsearch.cluster.ClusterService
> for parameter 2 at 
> org.elasticsearch.action.admin.indices.open.TransportOpenIndexAction.(Unknown
>  
> Source)
>   while locating 
> org.elasticsearch.action.admin.indices.open.TransportOpenIndexAction
>   while locating org.elasticsearch.action.support.TransportAction 
> annotated with 
> @org.elasticsearch.common.inject.multibindings.Element(setName=,uniqueId=305)
>   at _unknown_
>   while locating java.util.Map org.elasticsearch.action.support.TransportAction>
> for parameter 1 at 
> org.elasticsearch.client.node.NodeClusterAdminClient.(Unknown Source)
>   while locating org.elasticsearch.client.node.NodeClusterAdminClient
> for parameter 1 at 
> org.elasticsearch.client.node.NodeAdminClient.(Unknown Source)
>   while locating org.elasticsearch.client.node.NodeAdminClient
> for parameter 2 at 
> org.elasticsearch.client.node.NodeClient.(Unknown Source)
>   while locating org.elasticsearch.client.node.NodeClient
>   while locating org.elasticsearch.client.Client
> for parameter 2 at com.tr.es.plugin.TestESRestHandler.(Unknown 
> Source)
>   while locating com.tr.es.plugin.TestESRestHandler
>
> 2) Error injecting constructor, java.lang.IllegalStateException: This is a 
> proxy used to support circular references involving constructors. The 
> object we're proxying is not constructed yet. Please wait until after 
> injection has completed to use this object.
>   at org.elasticsearch.cluster.InternalClusterInfoService.(Unknown 
> Source)
>   while locating org.elasticsearch.cluster.InternalClusterInfoService
>   while locating org.elasticsearch.cluster.ClusterInfoService
> for parameter 3 at 
> org.elasticsearch.cluster.routing.allocation.AllocationService.(Unknown 
> Source)
>   while locating 
> org.elasticsearch.cluster.routing.allocation.AllocationService
> for parameter 2 at   
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5caebd06-cda7-47ee-95bc-42a24d6446bc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Transport error 404

[James]

Hi,

I'm trying to get elasticsearch working, querying it from python. However in my 
UWSGI logs I have this error:

File 
"/var/www/.../venv/local/lib/python2.7/site-packages/elasticsearch/connection/base.py",
 line 9
7, in _raise_error  
  
raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, 
error_message, additional_info)   
elasticsearch.exceptions.NotFoundError: TransportError(404, 
u'IndexMissingException[[employ_simple] missin
g]')

Is this a simple error just to say I don't have an index on my elasticsearch 
server yet? Do I have to create one before I can attempt to put any data on it?

James

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/2eac9753-9ea4-48cb-b8e1-7723fb63b56b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: percolator filtering not working?

[James Addison]

Ah! https://github.com/elasticsearch/elasticsearch/issues/6241, your very 
own bug submission, Martijn! I suspect this is the culprit - it's been an 
open issue for some time. At least I know it's not my fault... at the 
moment. :)

James

On Wednesday, February 4, 2015 at 9:19:11 AM UTC-8, James Addison wrote:
>
> Hi Martijn,
>
> Thanks for the reply - 'activities' is an alias to '
> activities-20150107-162224', and as I mentioned, percolation without 
> percolator filtering works fine - so this should mean that the percolator 
> **is** in the index and is accessible via the alias.
>
> So again, I'm not sure why percolator filtering isn't working - as I've 
> said, I assume I'm misunderstanding/doing something wrong, but I can't 
> figure out just what.
>
> James
>
> On Tuesday, February 3, 2015 at 10:43:14 PM UTC-8, Martijn v Groningen 
> wrote:
>>
>> The index that the percolator query is in is named: '
>> activities-20150107-162224', but your are percolating on the 
>> `activities` index. I think in the activities index there is no `doc_type` 
>> field defined in the `.percolator` type, so that is why there are no 
>> matches. If you you use activities* as index in the percolator call, 
>> matches should return.
>>
>> On 3 February 2015 at 20:50, James Addison  wrote:
>>
>>> Nobody knows anything about percolators?
>>>
>>> On Friday, January 30, 2015 at 2:32:33 PM UTC-8, James Addison wrote:
>>>>
>>>> Percolation works fine, as long as I don't 'filter' the percolators. 
>>>> Please see below for the scenario. What am I doing wrong?
>>>>
>>>>
>>>>
>>>>
>>>> percolator mapping:
>>>>
>>>> {
>>>>"activities-20150107-162224": {
>>>>   "mappings": {
>>>>  ".percolator": {
>>>> "_id": {
>>>>"index": "not_analyzed"
>>>> },
>>>> "properties": {
>>>>"doc_type": {
>>>>   "type": "string"
>>>>},
>>>>"query": {
>>>>   "type": "object",
>>>>   "enabled": false
>>>>}
>>>> }
>>>>  }
>>>>   }
>>>>}
>>>> }
>>>>
>>>> percolator query doc:
>>>>
>>>> {
>>>>"_index": "activities-20150107-162224",
>>>>"_type": ".percolator",
>>>>"_id": "tagrule-1",
>>>>"_version": 2,
>>>>"found": true,
>>>>"_source": {
>>>>   "query": {
>>>>  "query_string": {
>>>> "fields": [
>>>>"name",
>>>>"description"
>>>> ],
>>>> "query": "hockey"
>>>>  }
>>>>   },
>>>>   "doc_type": "activity"
>>>>}
>>>> }
>>>>
>>>> percolate request that works fine:
>>>>
>>>> GET /activities/activity/_percolate?percolate_format=ids
>>>> {
>>>> "doc": {
>>>> "name": "hockey"
>>>> }
>>>> }
>>>>
>>>> filtered percolate request that does NOT work.
>>>>
>>>> GET /activities/activity/_percolate?percolate_format=ids
>>>> {
>>>> "doc": {
>>>> "name": "hockey"
>>>> },
>>>> "query": {
>>>> "filtered": {
>>>> "query": {"match_all": {}}, 
>>>> "filter": {
>>>> "term": {
>>>>"doc_type": "activity"
>>>> }
>>>> }
>>>> }
>>>> }
>>>> }
>>>>
>>>> This also does not work:
>>>>
>>>> GET /activities/activity/_percolate?percolate_format=ids
>>>> {
>>>> "doc": {
>>>> "name": "hockey"
>>>> },
>>>> "filter": {
>>>> "term": {
>>>>"doc_type": "activity"
>>>> }
>>>> }
>>>> }
>>>>
>>>> I have done a /_refresh on the index. ES version is 1.4.2.
>>>>
>>>>  

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/44cf7fef-4daa-4086-b327-545af7e689bc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: percolator filtering not working?

[James Addison]

Hi Martijn,

Thanks for the reply - 'activities' is an alias to '
activities-20150107-162224', and as I mentioned, percolation without 
percolator filtering works fine - so this should mean that the percolator 
**is** in the index and is accessible via the alias.

So again, I'm not sure why percolator filtering isn't working - as I've 
said, I assume I'm misunderstanding/doing something wrong, but I can't 
figure out just what.

James

On Tuesday, February 3, 2015 at 10:43:14 PM UTC-8, Martijn v Groningen 
wrote:
>
> The index that the percolator query is in is named: '
> activities-20150107-162224', but your are percolating on the `activities` 
> index. I think in the activities index there is no `doc_type` field defined 
> in the `.percolator` type, so that is why there are no matches. If you you 
> use activities* as index in the percolator call, matches should return.
>
> On 3 February 2015 at 20:50, James Addison 
> > wrote:
>
>> Nobody knows anything about percolators?
>>
>> On Friday, January 30, 2015 at 2:32:33 PM UTC-8, James Addison wrote:
>>>
>>> Percolation works fine, as long as I don't 'filter' the percolators. Please 
>>> see below for the scenario. What am I doing wrong?
>>>
>>>
>>>
>>>
>>> percolator mapping:
>>>
>>> {
>>>"activities-20150107-162224": {
>>>   "mappings": {
>>>  ".percolator": {
>>> "_id": {
>>>"index": "not_analyzed"
>>> },
>>> "properties": {
>>>"doc_type": {
>>>   "type": "string"
>>>},
>>>"query": {
>>>   "type": "object",
>>>   "enabled": false
>>>}
>>> }
>>>  }
>>>   }
>>>}
>>> }
>>>
>>> percolator query doc:
>>>
>>> {
>>>"_index": "activities-20150107-162224",
>>>"_type": ".percolator",
>>>"_id": "tagrule-1",
>>>"_version": 2,
>>>"found": true,
>>>"_source": {
>>>   "query": {
>>>  "query_string": {
>>> "fields": [
>>>"name",
>>>"description"
>>> ],
>>> "query": "hockey"
>>>  }
>>>   },
>>>   "doc_type": "activity"
>>>}
>>> }
>>>
>>> percolate request that works fine:
>>>
>>> GET /activities/activity/_percolate?percolate_format=ids
>>> {
>>> "doc": {
>>> "name": "hockey"
>>> }
>>> }
>>>
>>> filtered percolate request that does NOT work.
>>>
>>> GET /activities/activity/_percolate?percolate_format=ids
>>> {
>>> "doc": {
>>> "name": "hockey"
>>> },
>>> "query": {
>>> "filtered": {
>>> "query": {"match_all": {}}, 
>>> "filter": {
>>> "term": {
>>>"doc_type": "activity"
>>> }
>>> }
>>> }
>>> }
>>> }
>>>
>>> This also does not work:
>>>
>>> GET /activities/activity/_percolate?percolate_format=ids
>>> {
>>> "doc": {
>>> "name": "hockey"
>>> },
>>> "filter": {
>>> "term": {
>>>"doc_type": "activity"
>>> }
>>> }
>>> }
>>>
>>> I have done a /_refresh on the index. ES version is 1.4.2.
>>>
>>>  

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5f4a6319-ed02-4673-8b4f-b7d2966a248c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: percolator filtering not working?

[James Addison]

Nobody knows anything about percolators?

On Friday, January 30, 2015 at 2:32:33 PM UTC-8, James Addison wrote:
>
> Percolation works fine, as long as I don't 'filter' the percolators. Please 
> see below for the scenario. What am I doing wrong?
>
>
>
>
> percolator mapping:
>
> {
>"activities-20150107-162224": {
>   "mappings": {
>  ".percolator": {
> "_id": {
>"index": "not_analyzed"
> },
> "properties": {
>"doc_type": {
>   "type": "string"
>},
>"query": {
>   "type": "object",
>   "enabled": false
>}
> }
>  }
>   }
>}
> }
>
> percolator query doc:
>
> {
>"_index": "activities-20150107-162224",
>"_type": ".percolator",
>"_id": "tagrule-1",
>"_version": 2,
>"found": true,
>"_source": {
>   "query": {
>  "query_string": {
> "fields": [
>"name",
>"description"
> ],
> "query": "hockey"
>  }
>   },
>   "doc_type": "activity"
>}
> }
>
> percolate request that works fine:
>
> GET /activities/activity/_percolate?percolate_format=ids
> {
> "doc": {
> "name": "hockey"
> }
> }
>
> filtered percolate request that does NOT work.
>
> GET /activities/activity/_percolate?percolate_format=ids
> {
> "doc": {
> "name": "hockey"
> },
> "query": {
> "filtered": {
> "query": {"match_all": {}}, 
> "filter": {
> "term": {
>"doc_type": "activity"
> }
> }
> }
> }
> }
>
> This also does not work:
>
> GET /activities/activity/_percolate?percolate_format=ids
> {
> "doc": {
> "name": "hockey"
> },
> "filter": {
> "term": {
>"doc_type": "activity"
> }
> }
> }
>
> I have done a /_refresh on the index. ES version is 1.4.2.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/29e62092-3023-4acc-951b-78d892cefb30%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

percolator filtering not working?

[James Addison]

Percolation works fine, as long as I don't 'filter' the percolators. Please see 
below for the scenario. What am I doing wrong?




percolator mapping:

{
   "activities-20150107-162224": {
  "mappings": {
 ".percolator": {
"_id": {
   "index": "not_analyzed"
},
"properties": {
   "doc_type": {
  "type": "string"
   },
   "query": {
  "type": "object",
  "enabled": false
   }
}
 }
  }
   }
}

percolator query doc:

{
   "_index": "activities-20150107-162224",
   "_type": ".percolator",
   "_id": "tagrule-1",
   "_version": 2,
   "found": true,
   "_source": {
  "query": {
 "query_string": {
"fields": [
   "name",
   "description"
],
"query": "hockey"
 }
  },
  "doc_type": "activity"
   }
}

percolate request that works fine:

GET /activities/activity/_percolate?percolate_format=ids
{
"doc": {
"name": "hockey"
}
}

filtered percolate request that does NOT work.

GET /activities/activity/_percolate?percolate_format=ids
{
"doc": {
"name": "hockey"
},
"query": {
"filtered": {
"query": {"match_all": {}}, 
"filter": {
"term": {
   "doc_type": "activity"
}
}
}
}
}

This also does not work:

GET /activities/activity/_percolate?percolate_format=ids
{
"doc": {
"name": "hockey"
},
"filter": {
"term": {
   "doc_type": "activity"
}
}
}

I have done a /_refresh on the index. ES version is 1.4.2.


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/9746982d-cdac-474e-8b0d-40996bbb3c3e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Accessing Port 9200 Remotely

[James Carr]

You should bind to 0.0.0.0 not localhost. Then you can hit it remotely.

You can also use a reverse-proxy. It is what we do :-)

On Mon, Jan 26, 2015 at 6:16 PM, Dave Mittner 
wrote:

> I'm binding to "localhost"; nothing else seems to work. I get "Failed to
> bind" errors in the log if I try using the actual hostname.
>
> On Monday, January 26, 2015 at 5:04:34 PM UTC-7, James Carr wrote:
>>
>> * make sure you haven't bound to 127.0.0.1
>> * make sure relevant security groups have been opened.
>>
>> On Mon, Jan 26, 2015 at 5:58 PM, Mark Walkom  wrote:
>>
>>> You don't need the plugin, it can make discovery easier though.
>>>
>>> You haven't bound ES to localhost only have you?
>>>
>>> On 27 January 2015 at 09:42, Dave Mittner  wrote:
>>>
>>>> I'm at a bit of a loss here...
>>>>
>>>> I have a pretty standard setup of elasticsearch and I'm able to
>>>> curl/telnet to port 9200 locally, but the connection is refused if accessed
>>>> from another server. I'm likewise unable to connect to port 9300. For
>>>> Kibana3 I've also added:
>>>>
>>>> http.cors.allow-origin: "*"
>>>> http.cors.enabled: true
>>>>
>>>> Our servers are on Amazon EC2 but we really don't need all the
>>>> automated discovery stuff within the EC2 plugin. The ports are open through
>>>> the security group and there's nothing in IP tables or any other firewalls.
>>>> I also get slightly different behavior once the ports were open so it feels
>>>> like the connection is being refused by the server or elasticsearch, 
>>>> itself.
>>>>
>>>> Any pointers?
>>>> Is the EC2 plugin a requirement in this case?
>>>>
>>>> Thanks
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "elasticsearch" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to elasticsearc...@googlegroups.com.
>>>> To view this discussion on the web visit https://groups.google.com/d/
>>>> msgid/elasticsearch/f8ecd552-336a-408b-9068-6d087d4c48e0%
>>>> 40googlegroups.com
>>>> <https://groups.google.com/d/msgid/elasticsearch/f8ecd552-336a-408b-9068-6d087d4c48e0%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>>  --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/elasticsearch/CAEYi1X84jXq9GK3AfYjxBNLsoJDJk
>>> sAUZSr-PvNaH%2BBZDkn3gg%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/elasticsearch/CAEYi1X84jXq9GK3AfYjxBNLsoJDJksAUZSr-PvNaH%2BBZDkn3gg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/b05ca44e-a4ad-482c-9f4c-53f845303d9f%40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/b05ca44e-a4ad-482c-9f4c-53f845303d9f%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAJreXKCPhwY7OYpKnOo6ORe5BucnyGMCyDWFgcQHxwB%2BxAig1w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Accessing Port 9200 Remotely

[James Carr]

* make sure you haven't bound to 127.0.0.1
* make sure relevant security groups have been opened.

On Mon, Jan 26, 2015 at 5:58 PM, Mark Walkom  wrote:

> You don't need the plugin, it can make discovery easier though.
>
> You haven't bound ES to localhost only have you?
>
> On 27 January 2015 at 09:42, Dave Mittner  wrote:
>
>> I'm at a bit of a loss here...
>>
>> I have a pretty standard setup of elasticsearch and I'm able to
>> curl/telnet to port 9200 locally, but the connection is refused if accessed
>> from another server. I'm likewise unable to connect to port 9300. For
>> Kibana3 I've also added:
>>
>> http.cors.allow-origin: "*"
>> http.cors.enabled: true
>>
>> Our servers are on Amazon EC2 but we really don't need all the automated
>> discovery stuff within the EC2 plugin. The ports are open through the
>> security group and there's nothing in IP tables or any other firewalls. I
>> also get slightly different behavior once the ports were open so it feels
>> like the connection is being refused by the server or elasticsearch, itself.
>>
>> Any pointers?
>> Is the EC2 plugin a requirement in this case?
>>
>> Thanks
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/f8ecd552-336a-408b-9068-6d087d4c48e0%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAEYi1X84jXq9GK3AfYjxBNLsoJDJksAUZSr-PvNaH%2BBZDkn3gg%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAJreXKDEaW6AbAHFo4YPn2gVHhK1AEa_rS-BdtvPdCh_eXkGnw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: need some n00b help: how do I install and set up synonyms file?

[james rubinstein]

Thanks Mark, 
I believe that I have done that (but perhaps not)

My file is located at

/Users/jrubinstein/dev/elasticsearch/elasticsearch-1.4.1/config/analysis

the YML files are in the 'config' directory. 


On Thursday, January 22, 2015 at 3:37:39 PM UTC-8, Mark Walkom wrote:
>
> The path needs to be relative to the config home for ES.
>
> Take a look at 
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-dir-layout.html#default-paths
>  
> for where that would be on your installation.
>
> On 23 January 2015 at 08:45, james rubinstein  > wrote:
>
>> Also, I tried this:
>>
>>
>>
>>
>>
>>
>>
>>
>> curl -XPUT 'localhost:9200/test/_settings' -d '
>>
>> > {
>>
>> > "index" : {
>>
>> > "analysis" : {
>>
>> > "analyzer" : {
>>
>> > "synonym" : {
>>
>> > "tokenizer" : "whitespace",
>>
>> > "filter" : ["synonym"]
>>
>> > }
>>
>> > },
>>
>> > "filter" : {
>>
>> > "synonym" : {
>>
>> > "type" : "synonym",
>>
>> > "synonyms_path" : "analysis/synonym.txt"
>>
>> > }
>>
>> > }
>>
>> > }
>>
>> > }
>>
>> > }'
>>
>> Which gives the response : {"acknowledged":true} , but it still didn't 
>> change the outcome when querying 
>>
>>
>> On Thursday, January 22, 2015 at 12:57:51 PM UTC-8, james rubinstein 
>> wrote:
>>>
>>> Hi all,
>>> I'm new to using ES, but have so far found the process quite intuitive. 
>>> I've installed ES, set up an index, indexed a bunch of JSON documents, and 
>>> I can search them. Hooray! 
>>> However, I want to start using synonyms for a few queries. I'd like to 
>>> do this at query time using a synonym file. How can I install the synonym 
>>> file and have ES read it? 
>>> I'm using elasticsearch-1.4.1 on Mac OSX locally. 
>>>
>>> I've read through the docs  
>>> <http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-update-settings.html>
>>>  but 
>>> I still don't understand what to do because I'm still not seeing queries 
>>> that use the synonym I've defined. 
>>>
>>>
>>>
>>>
>>> POST /test/_settings
>>> { 
>>> "index" : {
>>> "analysis" : {
>>> "analyzer" : {
>>> "synonym" : {
>>> "tokenizer" : "whitespace",
>>> "filter" : ["synonym"]
>>> }
>>> },
>>> "filter" : {
>>> "synonym" : {
>>> "type" : "synonym",
>>> "synonyms_path" : "analysis/synonym.txt"
>>> }
>>> }
>>> }
>>> }
>>> }
>>>
>>> My synonym file has one line:
>>>
>>> "clementine=> clementine,mandarin,orange,citrus"
>>>
>>> When I search for "Clementine" on my index I get 3 results, searching 
>>> for "Orange" gets me 66. 
>>>
>>> Thanks,
>>> JR
>>>
>>>
>>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/a887f420-b04b-4d5d-80aa-14c120fb9da3%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/elasticsearch/a887f420-b04b-4d5d-80aa-14c120fb9da3%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/3c71663d-91d3-4bc2-861f-c2e156db34dc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: need some n00b help: how do I install and set up synonyms file?

[james rubinstein]

Also, I tried this:








curl -XPUT 'localhost:9200/test/_settings' -d '

> {

> "index" : {

> "analysis" : {

> "analyzer" : {

> "synonym" : {

> "tokenizer" : "whitespace",

> "filter" : ["synonym"]

> }

> },

> "filter" : {

> "synonym" : {

> "type" : "synonym",

> "synonyms_path" : "analysis/synonym.txt"

> }

>     }

> }

> }

> }'

Which gives the response : {"acknowledged":true} , but it still didn't 
change the outcome when querying 

On Thursday, January 22, 2015 at 12:57:51 PM UTC-8, james rubinstein wrote:
>
> Hi all,
> I'm new to using ES, but have so far found the process quite intuitive. 
> I've installed ES, set up an index, indexed a bunch of JSON documents, and 
> I can search them. Hooray! 
> However, I want to start using synonyms for a few queries. I'd like to do 
> this at query time using a synonym file. How can I install the synonym file 
> and have ES read it? 
> I'm using elasticsearch-1.4.1 on Mac OSX locally. 
>
> I've read through the docs  
> <http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-update-settings.html>
>  but 
> I still don't understand what to do because I'm still not seeing queries 
> that use the synonym I've defined. 
>
>
>
>
> POST /test/_settings
> { 
> "index" : {
> "analysis" : {
> "analyzer" : {
> "synonym" : {
> "tokenizer" : "whitespace",
> "filter" : ["synonym"]
> }
> },
> "filter" : {
> "synonym" : {
> "type" : "synonym",
> "synonyms_path" : "analysis/synonym.txt"
> }
> }
> }
> }
> }
>
> My synonym file has one line:
>
> "clementine=> clementine,mandarin,orange,citrus"
>
> When I search for "Clementine" on my index I get 3 results, searching for 
> "Orange" gets me 66. 
>
> Thanks,
> JR
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/a887f420-b04b-4d5d-80aa-14c120fb9da3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

need some n00b help: how do I install and set up synonyms file?

[james rubinstein]

Hi all,
I'm new to using ES, but have so far found the process quite intuitive. 
I've installed ES, set up an index, indexed a bunch of JSON documents, and 
I can search them. Hooray! 
However, I want to start using synonyms for a few queries. I'd like to do 
this at query time using a synonym file. How can I install the synonym file 
and have ES read it? 
I'm using elasticsearch-1.4.1 on Mac OSX locally. 

I've read through the docs  

 but 
I still don't understand what to do because I'm still not seeing queries 
that use the synonym I've defined. 




POST /test/_settings
{ 
"index" : {
"analysis" : {
"analyzer" : {
"synonym" : {
"tokenizer" : "whitespace",
"filter" : ["synonym"]
}
},
"filter" : {
"synonym" : {
"type" : "synonym",
"synonyms_path" : "analysis/synonym.txt"
}
}
}
}
}

My synonym file has one line:

"clementine=> clementine,mandarin,orange,citrus"

When I search for "Clementine" on my index I get 3 results, searching for 
"Orange" gets me 66. 

Thanks,
JR


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d9043a12-8cd3-47fa-9800-5041215a6816%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Badly corrupted index

[James Green]

Our storage server threw it's toys of the pram. We got it back, and
everything bar ElasticSearch is singing sweetly again.

On start, we get an IndexFormatTooNewException. Not many get that one, but
CheckIndex appears to be worth a go, except that also barfs:

nicks@elasticsearch:/var/lib/elasticsearch/elasticsearch/nodes/0/indices/support/3/index$
sudo java -cp /usr/share/elasticsearch/lib/lucene-core-4.8.1.jar
-ea:org.apache.lucene... org.apache.lucene.index.CheckIndex . -fix
Opening index @ .
ERROR: could not read any segments file in directory
org.apache.lucene.index.IndexFormatTooNewException: Format version is not
supported (resource:
BufferedChecksumIndexInput(MMapIndexInput(path="/var/lib/elasticsearch/elasticsearch/nodes/0/indices/support/3/index/segments.gen"))):
1071082519 (needs to be between -2 and -3)
at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:803)
at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:710)
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:412)
at
org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:1967)

This has rendered our ES service offline. Any suggestions to fix it would
be appreciated, we are uncertain how to proceed otherwise.

ES 1.2.1 from Debian packages. We are just (now) upgrading to 1.4 to see if
this results in some progress.

James

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAMH6%2BaxYVvHEnjLocbeHh9v5pjReZ0aDds0aOtkjy1sDGmfY1w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Is it advisable NOT to use ES cluster for production?

[James Carr]

Depends on what your definition of "ok for production" is. For me, that
means sharding and replicas for appropriate failover when a node goes down.
I guess you're okay if you run a single server, have a backup / restore
process in place and searching be unavailable doesn't impact your
application heavily.


On Fri, Dec 26, 2014 at 8:54 PM, cordyceps  wrote:

> Hi, i'm wondering if running ES in non-cluster is ok for production, i
> have only < 1 million docs
> Please give me advice thanks.
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/261a6ea2-82c0-4552-9bf0-46da54db07ad%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAJreXKDhUeWyuZN6epuWfU0kGE2iYB03JoqC7wde-CjRD3LvnQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Progress on Hive "Push Down Filtering"

[James Andrew-Smith]

Hi Costin,

Thank you for the rapid response - just wanted to say I appreciate the 
Hadoop install works so easily just as advertised.

Shame about the push down filter but this is what I expected.

I'll focus on keeping the projection as lightweight as possible - on that 
note - I started another thread (
https://groups.google.com/forum/m/#!topic/elasticsearch/-3Lbdw5Wigg) about 
using aggregation queries (which I am a huge fan of) via Hive. Is this in 
possible/in the pipeline?

Cheers
James

On Thursday, 4 December 2014 20:04:17 UTC+11, Costin Leau wrote:
>
> Hi, 
>
> There are two aspects when dealing with large tables. 
>
> 1. Projection 
>
> The table mapping/definition is necessary as it indicates what information 
> is needed - a small mapping excludes a lot of 
> unnecessary data. 
>
> 2. Push Down filtering 
>
> Unfortunately there hasn't been much happening on this front since the 
> functionality is fairly restricted and not really 
> pluggable especially when 
> dealing with non-HDFS resources. The ORC support has improved things a bit 
> however it's still early days... 
>
> Cheers, 
>
>
> On 12/4/14 7:41 AM, James Andrew-Smith wrote: 
> > Has there been any progress on the "Push Down Filtering" mentioned by 
> Costin? 
> > (
> http://ryrobes.com/systems/connecting-tableau-to-elasticsearch-read-how-to-query-elasticsearch-with-hive-sql-and-hadoop/#comment-1169375542)
>  
>
> > 
> > 
> > Right now I am working around this by creating a lot of specific table 
> mappings to maintain performance. 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups "elasticsearch" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to 
> > elasticsearc...@googlegroups.com   elasticsearch+unsubscr...@googlegroups.com >. 
> > To view this discussion on the web visit 
> > 
> https://groups.google.com/d/msgid/elasticsearch/bd8151a0-af89-4617-9efe-aa738e70862c%40googlegroups.com
>  
> > <
> https://groups.google.com/d/msgid/elasticsearch/bd8151a0-af89-4617-9efe-aa738e70862c%40googlegroups.com?utm_medium=email&utm_source=footer>.
>  
>
> > For more options, visit https://groups.google.com/d/optout. 
>
> -- 
> Costin 
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/612d65f5-90f9-4ac5-93ca-5cf8fe6fb03f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Progress on Hive "Push Down Filtering"

[James Andrew-Smith]

Has there been any progress on the "Push Down Filtering" mentioned by 
Costin? (
http://ryrobes.com/systems/connecting-tableau-to-elasticsearch-read-how-to-query-elasticsearch-with-hive-sql-and-hadoop/#comment-1169375542
) 

Right now I am working around this by creating a lot of specific table 
mappings to maintain performance.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/bd8151a0-af89-4617-9efe-aa738e70862c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Using Hadoop/Hive With Aggregations

[James Andrew-Smith]

I want to expose some aggregate SUM/MIN/MAX data from ES through to my Hive 
tables. 

Currently I have created an table which needs to scroll through all the 
rows and then re-aggregate these statistics in Hive. 

This feels very inefficient. Is it possible to create an ES query with 
Aggregations as a table and have the aggregations/buckets come through to 
hive? 

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/03e31a7f-0cb0-49e7-978b-bf870354334d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

ElasticSearch.js some how hangs?

[James]

Hi all,

I'm writing a web crawler in Node.js and indexing with ElasticSearch. 
However, I've ran into a problem where the code hangs at the indexing 
function.

Here's how the client is initialised:

var es_client = new elasticsearch.Client({
host: "localhost:9200",
log: ['error', 'trace'],
keepAlive: true,
sniffOnConnectionFault: true,
//sniffInterval: 6000,
sniffOnStart: true,
maxKeepAliveTime: 60
});

And here's the indexing API call:

es_client.index({
  index: seedURL,
  type: 'post',
  id: generate_md5(username + "\n" + post_title + "\n" + post_content),
  body: {
thread_md5 : thread_md5,
thread_title : thread_title,
thread_url : post_list_page_url,
post_title: post_title,
post_order : post_order,
post_content: post_content,
timestamp: timestamp,
username: username,

  }
}).then(
function (resp) {
console.log("Elasticsearch response to indexing " + post_title 
+ "...");
console.log(resp);
}, 
function (err) {
console.log("[ERROR] An error occurred whilst indexing: " + 
post_title + "...");
console.log(err.message);
}
);

I have been testing the crawler script by commenting out the call to 
indexing and it finishes the crawl no problem. This showed that the problem 
somehow lies with ElasticSearch.

I have also had a look at the ElasticSearch logs and no errors were raised.

Lastly - and this could be the best hint yet - is that the number of 
documents successfully indexed at every trial run hangs at *exactly* 277 
documents. 

Thoughts?

Cheers,

James



-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/47d54f06-1ed8-4170-a019-31e88009fb06%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

How to get term vectors of all documents for a given type !!!

[James Parker]

Hi all, 

Is there a way to retrieve term vectors of all documents for a given type 
 using Elasticsearch Java API. 

Thanks, 

Evans

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/114eecdd-3fcb-43f4-bced-32f81080aaa3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: SSL reverse proxy in front of api on a cluster

[James Bennet]

The inter-cluster comms would still need secured too, I guess, though it is 
behind a firewall, so basic auth is less of an issue.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/bf1f4b39-3906-4009-b282-011f73895f9d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

SSL reverse proxy in front of api on a cluster

[James Bennet]

Hello.

Right now ive got an ES cluster, and I would like it to use SSL for 
security reasons (nodes located across the internet from one another, 
remote api requests) . I have considered using nginx as a reverse proxy (I 
would also like to do HTTP basic auth) - But im not quite sure how that 
would work. I think this may be nontrivial as I cant just set network.host 
to localhost and then put an ssl proxy from https:whatever:9200 to 
http:localhost:9200, can I? As I think that will break the clustering?

Has anyone done such a thing? I see its possible for single node instances, 
but what about clusters? - Or am I confused, and the 9200 is only the 
remote API and it only uses 9300-9400 for clustering?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/cf5d4315-58e9-48cb-85f0-a8f2033e61cf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Multi Field Aggregation

[Alastair James]

Hmmm. I dont know much about logstash, but I suspect thats concatenating
the 3 values into one string and taking a hash of it This would allow
you to group by that exact set of 3 columns however my use case is that
I need to be able to group by and subset of columns, so this could not be
pre-defined in that way.

Al

On 19 October 2014 16:48, Artur Martins  wrote:

> I heard that it could be done with a fingerprint, but I don't know how to
> do this. It's in logstash.conf
>
> Have a look:
>
> Fingerprint the 3-tuple of source address, destination address,
> destination port
>
> if [SourceAddress] and [DestinationAddress] {
>   fingerprint {
> concatenate_sources => true
> method => "SHA1"
> key => "logstash"
> source => [ "SourceAddress", "DestinationAddress", "DestinationPort" ]
>   }
> }
>
> But what exactly will this do? What next?
> Hope you can understand this and help us both 😊
>
> Thanks
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/gVLNqArGvVA/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/005d8152-9ee0-49bb-a8d5-84ccb9634124%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Dr Alastair James
CTO Ometria.com
Skype: al.james

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAMuyCY-u%2B6A%3DRUB1420BQZLt440eAShhSMeiwPWLLJgtq-Bm%3Dg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Multi Field Aggregation

[Alastair James]

Hi there.

No, no solution yet. I suspect its not possible in ES as it stands. It may 
require a custom aggregation plugin to be written, or to use scripting.

On Saturday, 18 October 2014 00:44:48 UTC+1, Artur Martins wrote:
>
> Hello,
>
> I'm having the exact same problem. 
> Have you managed to find a solution?
>
> My thread is here: LINK 
> <https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/Oum03VSBzHQ>
>
> Thanks
>
> On Thursday, October 16, 2014 1:57:35 PM UTC+1, Alastair James wrote:
>>
>> Hi there.
>>
>> I am trying to create an aggregation that mimics the following SQL query:
>>
>> SELECT col1, col2, COUNT(*), SUM(metric) FROM table GROUP BY col1, col2 
>> ORDER BY SUM(metric) DESC
>>
>> On the face of it, I could create an terms aggregation for col1, add a 
>> terms aggregation for col2 inside it, and the metric aggregations inside 
>> that. I could then dynamically build the SQL result like grid and sort it 
>> myself. However this breaks down for large results set, or a paginated 
>> result set of a larger result. 
>>
>> The problem is that the ES aggregation system always returns the top N 
>> results for each parent and child bucket. Thus for each value of col1 I 
>> have N values of col2.
>>
>> What I really want is to consider all possible combinations of col1 and 
>> col2 in the same way as SQL does it and return the top N based on some 
>> other metric. E.g. in ES speak, a single aggregation where the keys are 
>> tuples of (col1, col2).
>>
>> I suppose one way would be to use a script terms aggregation to 
>> concatenate each value of col1 and col2, however thats going to be slow.
>>
>> Does anyone else have any ideas?
>>
>> Ideally there would be a tuple aggregation built in, e.g.:
>>
>> "my_agg":{
>>"tuple":{
>>   "fields":["col1","col2"]
>>}
>> }
>>
>> Would product keys that are objects like:
>>
>> {
>>"col1":"value1",
>>"col2":"value2"
>> }
>>
>> Does anyone know if this would be possible to write as a  plugin?
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/497c90a2-1625-440f-a004-383dd3872703%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Multi Field Aggregation

[Alastair James]

Hi there.

I am trying to create an aggregation that mimics the following SQL query:

SELECT col1, col2, COUNT(*), SUM(metric) FROM table GROUP BY col1, col2 
ORDER BY SUM(metric) DESC

On the face of it, I could create an terms aggregation for col1, add a 
terms aggregation for col2 inside it, and the metric aggregations inside 
that. I could then dynamically build the SQL result like grid and sort it 
myself. However this breaks down for large results set, or a paginated 
result set of a larger result. 

The problem is that the ES aggregation system always returns the top N 
results for each parent and child bucket. Thus for each value of col1 I 
have N values of col2.

What I really want is to consider all possible combinations of col1 and 
col2 in the same way as SQL does it and return the top N based on some 
other metric. E.g. in ES speak, a single aggregation where the keys are 
tuples of (col1, col2).

I suppose one way would be to use a script terms aggregation to concatenate 
each value of col1 and col2, however thats going to be slow.

Does anyone else have any ideas?

Ideally there would be a tuple aggregation built in, e.g.:

"my_agg":{
   "tuple":{
  "fields":["col1","col2"]
   }
}

Would product keys that are objects like:

{
   "col1":"value1",
   "col2":"value2"
}

Does anyone know if this would be possible to write as a  plugin?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/9ec80e84-17b0-435d-a02e-f56a5d49f733%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: New version of Kibana in the works?

[James Green]

Those forking kibana on github and issuing a pull request - is their effort
in vain?

On 17 September 2014 23:37, Rashid Khan 
wrote:

> Unfortunately I can’t give you an ETA other than soon ;-)
>
> Initially backwards compatibility will not be available as there have been
> a large number of core changes. We’re looking at providing a compatibility
> layer to ease the transition, but there are some challenges there.
>
>
> On Thu, Aug 14, 2014 at 10:22 AM, Antonio Augusto Santos <
> mkha...@gmail.com> wrote:
>
>> Thanks a lot for the update Rashid.
>> I know it might be too much but... any dates we can mark on out calendar?
>> ;)
>>
>> On Thursday, August 14, 2014 1:04:24 PM UTC-3, Rashid Khan wrote:
>>>
>>> Yes there is a new version in the works.
>>>
>>> Elasticsearch has grown considerably in the last 18 months. With
>>> Elasticsearch 1.0, and the introduction of aggregations, we quickly
>>> realized we needed a better platform for building more powerful interfaces.
>>> So for the last few months we've been head down, working to polish Kibana 4
>>> to the point where we can collect feedback and contributions.
>>>
>>> We've left lots of pull requests open for a couple reasons: For the
>>> community to pull into their own fork if needed, and for our own
>>> inspiration. When someone submits code it says "I really want this!". Even
>>> if the code doesn't get merged, we take it very seriously. Once we push out
>>> the first preview of Kibana 4 we'll be cleaning up lots of pull requests
>>> for features we're implemented.
>>>
>>> As for the features you're looking for: User separation and event
>>> correlation are things we're looking to do in Elasticsearch. Multi-value
>>> metric aggregations are a feature I'm looking forward to in Kibana. It
>>> will, of course, be open source.
>>>
>>> On Tuesday, August 12, 2014 9:44:55 AM UTC-7, Antonio Augusto Santos
>>> wrote:

 This one is for the devs, and Rashid in special: there is any new
 version of Kibana in the works?
 I'm asking this because I'm about to start a project in my company for
 log management, and there are some requisites to it (user separation, event
 correlation, histogram to compare two values, and so on).

 So, any changes of these functionalities landing on Kibana 4.0? ;)

>>>--
>> You received this message because you are subscribed to a topic in the
>> Google Groups "elasticsearch" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/elasticsearch/I7um1mX4GSk/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/0a3bb3d3-c2dc-418b-bce0-5d376065fb7f%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/1410993466818.29294e6a%40Nodemailer
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAMH6%2Bawj49An6csfQfy8Sc34jihhDCH1x-s2OETzbqSXinJeiQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Sorting a random set of documents.

[James Moss]

Hi,

I need to pull out a random selection of documents (based on a seed) and 
then make that set sortable but I'm having difficulties:

Here's what I've got at the moment, I'm trying to pull out a random 
selection of document (real estate properties) and then sort them by price 
desc:

GET _search
{
"from": 0,
"size": 30,
"query": {
"function_score": {
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"and": [
{
"term": {
"property.rental": 0
}
},
{
"term": {
"property.status": 100
}
}
]
}
}
},
"random_score": {
"seed": 124567
}
}
},
"sort": [
{
"property.price": "desc"
}
],
"filter": {
"range": {
"property.price": {
"gte": 50
}
}
}
}

It seems the sort is being applied to all the documents in the index, not 
just those returned from the query.

Is this even possible with elasticsearch?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b82ba776-854b-4649-bd71-2719cf095061%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Range Aggregations: response bucket order?

[James Addison]

   

I would have thought that range aggregations return the bucket list in the 
order the range agg is listed, but I'm not seeing that result. (ES 1.3.2) 
Is there a way to enforce original ordering that I'm missing?
  My range aggregation:

"day_of_week_range": {
"range": {
"field": "sessions.events.days_of_week",
"ranges": [
{"from":0, "to":5, "key": "Weekdays"},
{"from":5, "to":7, "key": "Weekends"},
{"from":6, "to":6, "key": "Sunday"},
{"from":0, "to":1, "key": "Monday"},
{"from":1, "to":2, "key": "Tuesday"},
{"from":2, "to":3, "key": "Wednesday"},
{"from":3, "to":4, "key": "Thursday"},
{"from":4, "to":5, "key": "Friday"},
{"from":5, "to":6, "key": "Saturday"}
]
}
}

And the response (Note that Weekdays is 2nd in the response, which is why 
I'm posting):

"day_of_week_range": {
 "buckets": [
{
   "key": "Monday",
   "from": 0,
   "from_as_string": "0.0",
   "to": 1,
   "to_as_string": "1.0",
   "doc_count": 11
},
{
   "key": "Weekdays",
   "from": 0,
   "from_as_string": "0.0",
   "to": 5,
   "to_as_string": "5.0",
   "doc_count": 40
},
{
   "key": "Tuesday",
   "from": 1,
   "from_as_string": "1.0",
   "to": 2,
   "to_as_string": "2.0",
   "doc_count": 11
},
{
   "key": "Wednesday",
   "from": 2,
   "from_as_string": "2.0",
   "to": 3,
   "to_as_string": "3.0",
   "doc_count": 10
},
{
   "key": "Thursday",
   "from": 3,
   "from_as_string": "3.0",
   "to": 4,
   "to_as_string": "4.0",
   "doc_count": 17
},
{
   "key": "Friday",
   "from": 4,
   "from_as_string": "4.0",
   "to": 5,
   "to_as_string": "5.0",
   "doc_count": 15
},
{
   "key": "Saturday",
   "from": 5,
   "from_as_string": "5.0",
   "to": 6,
   "to_as_string": "6.0",
   "doc_count": 28
},
{
   "key": "Weekends",
   "from": 5,
   "from_as_string": "5.0",
   "to": 7,
   "to_as_string": "7.0",
   "doc_count": 37
},
{
   "key": "Sunday",
   "from": 6,
   "from_as_string": "6.0",
   "to": 6,
   "to_as_string": "6.0",
   "doc_count": 0
}
 ]
}

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/9a245439-0fd8-4dd3-888b-43ec882c0e16%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Aggregations: Possible to return additional fields?

[James Addison]

Is it possible to do a terms aggregation on a 'tags.id' field but also get 
back the associated 'tags.slug' and 'tags.name' fields? Here's how the 
documents are structured:

{
"id: 42,
"name": Beer,
"tags: [{
"id": 2,
"name": "Hoppy",
"slug": "hoppy"
}, {
"id": 2,
"name": "Hoppy",
"slug": "hoppy"
}]
}

Obviously, this is so that I can avoid additional database or elasticsearch 
queries in order to display UI navigation/filtering elements.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/0ac8182e-5233-4e6e-8939-fe19bb455561%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Annotating results?

[James Addison]

 

I have two types stored in an index: locations and activities.An activity 
has a 'relation' to a location - ie. an activity takes place at a location. 
Is it possible to get a location search result set that includes the count 
of activities at each location? Sort of like annotating each location in 
location results, I guess?

Thanks.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/fff4cf9c-a926-431b-86f7-ad1f9850f86e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Is it possible to annotate each result in a result set?

[James Addison]

 

I have two types stored in an index: locations and activities. An activity 
has a 'relation' to a location - ie. an activity takes place at a location.

Is it possible to get a location search result set that includes the count 
of activities at each location? Sort of like annotating each item in the 
location results?

(This is my second attempt to post - apologies if this is a duplicate post!)

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/c39df045-7570-4fb4-83d6-702b4891a503%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Do I need the JDBC driver

[James]

I want to close this issue but I still do not understand if I should be 
pushing documents from my database using the PHP client or using the JDBC 
river to pull them into elasticsearch from the SQL database. 

They can both achieve the same thing, but what is the usecase which defines 
when is the right time to use each implementation.



On Wednesday, September 10, 2014 10:59:18 AM UTC+1, James wrote:
>
> Hi,
>
> I'm setting up a system where I have a main SQL database which is synced 
> with elasticsearch. My plan is to use the main PHP library for 
> elasticsearch. 
>
> I was going to have a cron run every thirty minuets to check for items in 
> my database that not only have an "active" flag but that also do not have 
> an "indexed" flag, that means I need to add them to the index. Then I was 
> going to add that item to the index. Since I am using taking this path, it 
> doesn't seem like I need the JDBC driver, as I can add items to 
> elasticsearch using the PHP library.
>
> So, my question is, can I get away without using the JDBC driver?
>
> James
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/6c244e00-1f89-447d-8eb5-114f0b5efcbd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Do I need the JDBC driver

[James]

Hi Jorg,

Thank you for the reply. Yes I meant the elasticsearch river. Simply put, I 
want to syncronize the entries in my SQL database with my elasticsearch, so 
I can use elasicsearch for searching and not doing fulltext search. I want 
to know that when a new item gets added or removed from that database that 
it also gets added / removed from elasicsearch.

My understand, which might be wrong, is I can either use the PHP 
elasticsearch library to push updates (adds / removes) to elasticsearch 
when new items are added to SQL:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-index_.html

Or I can use the river JDBC river plugin for elasticsearch to connect to my 
database directly and syncronize elasticsearch with the SQL database.

My two questions are:

1. Is my understanding above correct
2. Does one option have advantages over the other

- James

On Wednesday, September 10, 2014 10:59:18 AM UTC+1, James wrote:
>
> Hi,
>
> I'm setting up a system where I have a main SQL database which is synced 
> with elasticsearch. My plan is to use the main PHP library for 
> elasticsearch. 
>
> I was going to have a cron run every thirty minuets to check for items in 
> my database that not only have an "active" flag but that also do not have 
> an "indexed" flag, that means I need to add them to the index. Then I was 
> going to add that item to the index. Since I am using taking this path, it 
> doesn't seem like I need the JDBC driver, as I can add items to 
> elasticsearch using the PHP library.
>
> So, my question is, can I get away without using the JDBC driver?
>
> James
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1d5fe901-fd0e-4663-9c68-5f7cf8092cf1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Do I need the JDBC driver

[James]

I'm sorry but that doesn't answer my question. It's elasticsearch that is
Java. I need to sync elasticsearch with my SQL DB. I'm stuck between these
two scenarios:

Scenario 1:
PHP website adds data to the SQL DB
JDBC driver used by elasticsearch to grab values from SQL DB into index

Scenario 2:
PHP website adds data to SQL
CRON job uses PHP elasticsearch library to convert SQL to JSON and send it
to elasticsearch to be indexed.

Which one is the better implementation?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAPng%3D3e2ykmb4S2aqaXiLbV1h39ZE_ZWvHkX4OmVEQV2hPK3_A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Do I need the JDBC driver

[James]

I've also put this question up on stackoverflow for anyone who might be 
able to help me understand.

http://stackoverflow.com/questions/25763997/elasticsearch-do-i-need-the-jdbc-driver

On Wednesday, September 10, 2014 10:59:18 AM UTC+1, James wrote:
>
> Hi,
>
> I'm setting up a system where I have a main SQL database which is synced 
> with elasticsearch. My plan is to use the main PHP library for 
> elasticsearch. 
>
> I was going to have a cron run every thirty minuets to check for items in 
> my database that not only have an "active" flag but that also do not have 
> an "indexed" flag, that means I need to add them to the index. Then I was 
> going to add that item to the index. Since I am using taking this path, it 
> doesn't seem like I need the JDBC driver, as I can add items to 
> elasticsearch using the PHP library.
>
> So, my question is, can I get away without using the JDBC driver?
>
> James
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/3d07653f-a62b-4e16-bab0-8a9b0008730d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Do I need the JDBC driver

[James]

I don't really see why though, can't the php library send JSON data to
elasticsearch?

On 10 September 2014 12:47, joergpra...@gmail.com 
wrote:

> You need JDBC (Java Database Connectivity) for Java.
>
> Jörg
>
> On Wed, Sep 10, 2014 at 1:30 PM, James  wrote:
>
>> It just seems I can send data to elasticsearch using the php library, so
>> I don't seen the need to add the JDBC driver?
>>
>>
>> http://www.elasticsearch.org/guide/en/elasticsearch/client/php-api/current/_indexing_operations.html
>>
>> Use a CRON operation to check for items that aren't indexed, and then use
>> the PHP library to send the non-indexed data to ES to get indexed?
>>
>> On Wednesday, September 10, 2014 12:21:35 PM UTC+1, vineeth mohan wrote:
>>>
>>> Hello James ,
>>>
>>> I didn't fully understand your question , but i feel JDBC river might be
>>> of any use to you - https://github.com/jprante/elasticsearch-river-jdbc
>>>
>>> Thanks
>>>Vineeth
>>>
>>> On Wed, Sep 10, 2014 at 3:29 PM, James  wrote:
>>>
>>>> Hi,
>>>>
>>>> I'm setting up a system where I have a main SQL database which is
>>>> synced with elasticsearch. My plan is to use the main PHP library for
>>>> elasticsearch.
>>>>
>>>> I was going to have a cron run every thirty minuets to check for items
>>>> in my database that not only have an "active" flag but that also do not
>>>> have an "indexed" flag, that means I need to add them to the index. Then I
>>>> was going to add that item to the index. Since I am using taking this path,
>>>> it doesn't seem like I need the JDBC driver, as I can add items to
>>>> elasticsearch using the PHP library.
>>>>
>>>> So, my question is, can I get away without using the JDBC driver?
>>>>
>>>> James
>>>>
>>>>
>>>>  --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "elasticsearch" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to elasticsearc...@googlegroups.com.
>>>> To view this discussion on the web visit https://groups.google.com/d/
>>>> msgid/elasticsearch/be682d05-bdad-45a4-8b00-2ecf26217534%
>>>> 40googlegroups.com
>>>> <https://groups.google.com/d/msgid/elasticsearch/be682d05-bdad-45a4-8b00-2ecf26217534%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>>  --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/a9901f1a-2753-46d2-9113-056b8d996eb8%40googlegroups.com
>> <https://groups.google.com/d/msgid/elasticsearch/a9901f1a-2753-46d2-9113-056b8d996eb8%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/0dzSMbARlks/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoF33zrKTRqFhr%3D%3DMG2fYFYd04kezBqEcoXgYrFuaEyT6w%40mail.gmail.com
> <https://groups.google.com/d/msgid/elasticsearch/CAKdsXoF33zrKTRqFhr%3D%3DMG2fYFYd04kezBqEcoXgYrFuaEyT6w%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAPng%3D3f6qkSEehuowD%3D1QVKnS2JgoGqqYjasL5gWsLkz%2Bq%3D29A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Do I need the JDBC driver

[James]

It just seems I can send data to elasticsearch using the php library, so I 
don't seen the need to add the JDBC driver?

http://www.elasticsearch.org/guide/en/elasticsearch/client/php-api/current/_indexing_operations.html

Use a CRON operation to check for items that aren't indexed, and then use 
the PHP library to send the non-indexed data to ES to get indexed?

On Wednesday, September 10, 2014 12:21:35 PM UTC+1, vineeth mohan wrote:
>
> Hello James , 
>
> I didn't fully understand your question , but i feel JDBC river might be 
> of any use to you - https://github.com/jprante/elasticsearch-river-jdbc
>
> Thanks
>Vineeth
>
> On Wed, Sep 10, 2014 at 3:29 PM, James > 
> wrote:
>
>> Hi,
>>
>> I'm setting up a system where I have a main SQL database which is synced 
>> with elasticsearch. My plan is to use the main PHP library for 
>> elasticsearch. 
>>
>> I was going to have a cron run every thirty minuets to check for items in 
>> my database that not only have an "active" flag but that also do not have 
>> an "indexed" flag, that means I need to add them to the index. Then I was 
>> going to add that item to the index. Since I am using taking this path, it 
>> doesn't seem like I need the JDBC driver, as I can add items to 
>> elasticsearch using the PHP library.
>>
>> So, my question is, can I get away without using the JDBC driver?
>>
>> James
>>
>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/be682d05-bdad-45a4-8b00-2ecf26217534%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/elasticsearch/be682d05-bdad-45a4-8b00-2ecf26217534%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/a9901f1a-2753-46d2-9113-056b8d996eb8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Do I need the JDBC driver

[James]

Hi,

I'm setting up a system where I have a main SQL database which is synced 
with elasticsearch. My plan is to use the main PHP library for 
elasticsearch. 

I was going to have a cron run every thirty minuets to check for items in 
my database that not only have an "active" flag but that also do not have 
an "indexed" flag, that means I need to add them to the index. Then I was 
going to add that item to the index. Since I am using taking this path, it 
doesn't seem like I need the JDBC driver, as I can add items to 
elasticsearch using the PHP library.

So, my question is, can I get away without using the JDBC driver?

James


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/be682d05-bdad-45a4-8b00-2ecf26217534%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: NullPointerException during snapshot

[Alastair James]

Wondering if anyone has any input on this?

On Monday, 8 September 2014 10:53:24 UTC+1, Alastair James wrote:
>
> Hi there.
>
> I recently had a snapshot (ec2 plugin to s3) fail on one shard with a 
> NullPointerException. Unfortunately I did not keep the exact error message.
>
> However, I tried the snapshot again and it worked fine. So my question is: 
> Is the snapshot data now in a consistent state? Did the one failed shard 
> snapshot cause me to loose data or did that shard catch up on next snapshot?
>
> Cheers
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/9a16c52e-cb3a-4c64-938a-e9f3e063f614%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Do I need a plugin to search geographically

[James]

Ah cool ok. And as a last question I should separate these out into two
indexes on elasticsearch, one for items and one for geocoding?

On 8 September 2014 14:12, Itamar Syn-Hershko  wrote:

> 1000 entries is a very small set. If you can have this local to your code
> that would be best. Otherwise yes a 2-phase query is probably your best bet
> - note you could also use the suggesters to improve speed
>
> --
>
> Itamar Syn-Hershko
> http://code972.com | @synhershko <https://twitter.com/synhershko>
> Freelance Developer & Consultant
> Author of RavenDB in Action <http://manning.com/synhershko/>
>
> On Mon, Sep 8, 2014 at 4:07 PM, James  wrote:
>
>> Thank you for the reply. It will be 1000 entries for the UK alone. Do I
>> need to have two indexes, one for geocoding and one for item search, and
>> then every time a location search is done I do two queries, the first to
>> get the co-ordinates and the second to search items with the co-ordinates
>>
>> On 8 September 2014 14:04, Itamar Syn-Hershko  wrote:
>>
>>> This is just a key/value lookup. If it's not too big I'd just hold it
>>> all in memory. Otherwise of course you can use Elasticsearch for that.
>>>
>>> --
>>>
>>> Itamar Syn-Hershko
>>> http://code972.com | @synhershko <https://twitter.com/synhershko>
>>> Freelance Developer & Consultant
>>> Author of RavenDB in Action <http://manning.com/synhershko/>
>>>
>>> On Mon, Sep 8, 2014 at 3:39 PM, James  wrote:
>>>
>>>> It's a bit hard to word this so please bear with me. I want my users to
>>>> be able to search by place-name, but elasticsearch needs co-ordinates. I
>>>> have a premade SQL schema with place-names associated with co-ordinates.
>>>>
>>>> How can I implement this from my PHP website?
>>>>
>>>> A) On form submit, get my PHP website to do an SQL lookup to get the
>>>> co-ordinate from the place-name and send the co-ordinate to elasticsearch.
>>>>
>>>> B) Import my place-name to co-ordinate database into elasticsearch,
>>>> then on form submit send the place-name to elasticsearch and have
>>>> elasticsearch do the conversion from place-name to location.
>>>>
>>>> I guess what I am saying is, can I use elasticsearch as a geocoder as
>>>> part of my search query or will it only accept co-ordinates?
>>>>
>>>> On 3 September 2014 22:52, Itamar Syn-Hershko 
>>>> wrote:
>>>>
>>>>> Yes
>>>>>
>>>>> --
>>>>>
>>>>> Itamar Syn-Hershko
>>>>> http://code972.com | @synhershko <https://twitter.com/synhershko>
>>>>> Freelance Developer & Consultant
>>>>> Author of RavenDB in Action <http://manning.com/synhershko/>
>>>>>
>>>>>
>>>>> On Thu, Sep 4, 2014 at 12:49 AM, Employ  wrote:
>>>>>
>>>>>> Thank you. And no plugin is required to filter by distance, this is
>>>>>> included in the core ES module?
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 3 Sep 2014, at 22:19, Itamar Syn-Hershko 
>>>>>> wrote:
>>>>>>
>>>>>> You don't need any external geo-data, see
>>>>>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-distance-filter.html
>>>>>>
>>>>>> You only need external data sources if you want to give some
>>>>>> coordinates / polygons names (like countries, neighborhoods etc)
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Itamar Syn-Hershko
>>>>>> http://code972.com | @synhershko <https://twitter.com/synhershko>
>>>>>> Freelance Developer & Consultant
>>>>>> Author of RavenDB in Action <http://manning.com/synhershko/>
>>>>>>
>>>>>>
>>>>>> On Wed, Sep 3, 2014 at 11:13 PM, James  wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I'm a bit confused with this. If have results in elasticsearch with
>>>>>>> lat and long coordinates, and I search using lat, long and distance
>>>>>>> parameters, can elastic search return all the results within that given
>>>>>>> area without me having to use any third party 

Re: Do I need a plugin to search geographically

[James]

Thank you for the reply. It will be 1000 entries for the UK alone. Do I
need to have two indexes, one for geocoding and one for item search, and
then every time a location search is done I do two queries, the first to
get the co-ordinates and the second to search items with the co-ordinates

On 8 September 2014 14:04, Itamar Syn-Hershko  wrote:

> This is just a key/value lookup. If it's not too big I'd just hold it all
> in memory. Otherwise of course you can use Elasticsearch for that.
>
> --
>
> Itamar Syn-Hershko
> http://code972.com | @synhershko <https://twitter.com/synhershko>
> Freelance Developer & Consultant
> Author of RavenDB in Action <http://manning.com/synhershko/>
>
> On Mon, Sep 8, 2014 at 3:39 PM, James  wrote:
>
>> It's a bit hard to word this so please bear with me. I want my users to
>> be able to search by place-name, but elasticsearch needs co-ordinates. I
>> have a premade SQL schema with place-names associated with co-ordinates.
>>
>> How can I implement this from my PHP website?
>>
>> A) On form submit, get my PHP website to do an SQL lookup to get the
>> co-ordinate from the place-name and send the co-ordinate to elasticsearch.
>>
>> B) Import my place-name to co-ordinate database into elasticsearch, then
>> on form submit send the place-name to elasticsearch and have elasticsearch
>> do the conversion from place-name to location.
>>
>> I guess what I am saying is, can I use elasticsearch as a geocoder as
>> part of my search query or will it only accept co-ordinates?
>>
>> On 3 September 2014 22:52, Itamar Syn-Hershko  wrote:
>>
>>> Yes
>>>
>>> --
>>>
>>> Itamar Syn-Hershko
>>> http://code972.com | @synhershko <https://twitter.com/synhershko>
>>> Freelance Developer & Consultant
>>> Author of RavenDB in Action <http://manning.com/synhershko/>
>>>
>>>
>>> On Thu, Sep 4, 2014 at 12:49 AM, Employ  wrote:
>>>
>>>> Thank you. And no plugin is required to filter by distance, this is
>>>> included in the core ES module?
>>>>
>>>>
>>>>
>>>> On 3 Sep 2014, at 22:19, Itamar Syn-Hershko  wrote:
>>>>
>>>> You don't need any external geo-data, see
>>>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-distance-filter.html
>>>>
>>>> You only need external data sources if you want to give some
>>>> coordinates / polygons names (like countries, neighborhoods etc)
>>>>
>>>> --
>>>>
>>>> Itamar Syn-Hershko
>>>> http://code972.com | @synhershko <https://twitter.com/synhershko>
>>>> Freelance Developer & Consultant
>>>> Author of RavenDB in Action <http://manning.com/synhershko/>
>>>>
>>>>
>>>> On Wed, Sep 3, 2014 at 11:13 PM, James  wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I'm a bit confused with this. If have results in elasticsearch with
>>>>> lat and long coordinates, and I search using lat, long and distance
>>>>> parameters, can elastic search return all the results within that given
>>>>> area without me having to use any third party plugins?
>>>>>
>>>>> Do I need any external geo-data, or is it enough that I have my
>>>>> co-ordinates already?
>>>>>
>>>>> James
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "elasticsearch" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to elasticsearch+unsubscr...@googlegroups.com.
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/elasticsearch/c4626888-0014-43fb-81d8-cc6298bfd827%40googlegroups.com
>>>>> <https://groups.google.com/d/msgid/elasticsearch/c4626888-0014-43fb-81d8-cc6298bfd827%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>
>>>>  --
>>>> You received this message because you are subscribed to a topic in the
>>>> Google Groups "elasticsearch" group.
>>>> To unsubscribe from this topic, visit
>>>> https://groups.google.com/d/topic/elasticsearch/yQI82QIdEq

Re: Do I need a plugin to search geographically

[James]

It's a bit hard to word this so please bear with me. I want my users to be
able to search by place-name, but elasticsearch needs co-ordinates. I have
a premade SQL schema with place-names associated with co-ordinates.

How can I implement this from my PHP website?

A) On form submit, get my PHP website to do an SQL lookup to get the
co-ordinate from the place-name and send the co-ordinate to elasticsearch.

B) Import my place-name to co-ordinate database into elasticsearch, then on
form submit send the place-name to elasticsearch and have elasticsearch do
the conversion from place-name to location.

I guess what I am saying is, can I use elasticsearch as a geocoder as part
of my search query or will it only accept co-ordinates?

On 3 September 2014 22:52, Itamar Syn-Hershko  wrote:

> Yes
>
> --
>
> Itamar Syn-Hershko
> http://code972.com | @synhershko <https://twitter.com/synhershko>
> Freelance Developer & Consultant
> Author of RavenDB in Action <http://manning.com/synhershko/>
>
>
> On Thu, Sep 4, 2014 at 12:49 AM, Employ  wrote:
>
>> Thank you. And no plugin is required to filter by distance, this is
>> included in the core ES module?
>>
>>
>>
>> On 3 Sep 2014, at 22:19, Itamar Syn-Hershko  wrote:
>>
>> You don't need any external geo-data, see
>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-distance-filter.html
>>
>> You only need external data sources if you want to give some coordinates
>> / polygons names (like countries, neighborhoods etc)
>>
>> --
>>
>> Itamar Syn-Hershko
>> http://code972.com | @synhershko <https://twitter.com/synhershko>
>> Freelance Developer & Consultant
>> Author of RavenDB in Action <http://manning.com/synhershko/>
>>
>>
>> On Wed, Sep 3, 2014 at 11:13 PM, James  wrote:
>>
>>> Hi,
>>>
>>> I'm a bit confused with this. If have results in elasticsearch with lat
>>> and long coordinates, and I search using lat, long and distance parameters,
>>> can elastic search return all the results within that given area without me
>>> having to use any third party plugins?
>>>
>>> Do I need any external geo-data, or is it enough that I have my
>>> co-ordinates already?
>>>
>>> James
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearch+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elasticsearch/c4626888-0014-43fb-81d8-cc6298bfd827%40googlegroups.com
>>> <https://groups.google.com/d/msgid/elasticsearch/c4626888-0014-43fb-81d8-cc6298bfd827%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "elasticsearch" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/elasticsearch/yQI82QIdEqU/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuyS053gYN-VK8ta1vzRH6vuBJAkQ%3D_XBw-y-gciXet4Q%40mail.gmail.com
>> <https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuyS053gYN-VK8ta1vzRH6vuBJAkQ%3D_XBw-y-gciXet4Q%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>>  --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/86F7DFA5-4F62-4861-BEAF-29EA81317FD2%40employ.com
>> <https://groups.google.com/d/msgid/elasticsearch/86F7DFA5-4F62-4861-BEAF-29EA81317FD2%40employ.com?utm_medium=email&utm_source=footer>
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this top

NullPointerException during snapshot

[Alastair James]

Hi there.

I recently had a snapshot (ec2 plugin to s3) fail on one shard with a 
NullPointerException. Unfortunately I did not keep the exact error message.

However, I tried the snapshot again and it worked fine. So my question is: 
Is the snapshot data now in a consistent state? Did the one failed shard 
snapshot cause me to loose data or did that shard catch up on next snapshot?

Cheers

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/8cfb63a6-6b7e-410d-bf98-c212a4d42517%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Do I need a plugin to search geographically

[James]

Hi,

I'm a bit confused with this. If have results in elasticsearch with lat and 
long coordinates, and I search using lat, long and distance parameters, can 
elastic search return all the results within that given area without me 
having to use any third party plugins?

Do I need any external geo-data, or is it enough that I have my 
co-ordinates already?

James

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/c4626888-0014-43fb-81d8-cc6298bfd827%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: EsRejectedExecutionException: rejected execution (queue capacity 1000)

[James]

On Mon, 2014-08-18 at 10:34 -0300, Sávio S. Teles de Oliveira wrote:
> You can put threadpool.search.type: cached on elasticsearch.yml for
> unbounded queue for reads.
> 
> 
> 
> 

Thank you...that helps as well.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1408533672.5479.20.camel%40JamesiMac.
For more options, visit https://groups.google.com/d/optout.

Re: New version of Kibana in the works?

[James Green]

If Kibana 4 is not using the existing code base then perhaps someone needs
to reject a whole pile of pull requests as they are a lot waiting.

Not quite the way I'd expect an open source project to progress either.


On 13 August 2014 16:33, Antonio Augusto Santos 
wrote:

> Thanks Jay!
> These are very interesting info.
>
> Hope something official come out. And that Kibana 4 is open source as
> well...
>
>
> On Wed, Aug 13, 2014 at 12:27 PM, Jay Swan  wrote:
>
>> When I took the Core Elasticsearch class last week, someone from
>> Elasticsearch said that Kibana 4.0 is in the works but is at least a couple
>> of months away from release. There wasn't any real detail on features,
>> other than to say that it is a complete rewrite that tries to expose as
>> much native Elasticsearch query functionality as possible.
>>
>> I was also told that most security-related features will be released in a
>> commercial "security module" for Elasticsearch that will have a non-trivial
>> cost associated with it. I don't know if this includes role-based access
>> control or not.
>>
>>
>>
>> On Tuesday, August 12, 2014 10:44:55 AM UTC-6, Antonio Augusto Santos
>> wrote:
>>>
>>> This one is for the devs, and Rashid in special: there is any new
>>> version of Kibana in the works?
>>> I'm asking this because I'm about to start a project in my company for
>>> log management, and there are some requisites to it (user separation, event
>>> correlation, histogram to compare two values, and so on).
>>>
>>> So, any changes of these functionalities landing on Kibana 4.0? ;)
>>>
>>  --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "elasticsearch" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/elasticsearch/I7um1mX4GSk/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/ca643d8f-2b03-4e4d-930c-5d9a0a6a70eb%40googlegroups.com
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAGz5QREtDi5QtBhpNpsE2aQGVh8NiydaT4Zq%3D8EQ9Zth_Vt-iQ%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAMH6%2Baw4QnkzB88P5U-h0tLiML7wmvr_we5YUZGGJnPtKoOhwA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: New version of Kibana in the works?

[James Green]

There's been nothing since 13 Jun according to Github. Most disappointing,
lots of people waiting for their PRs to be attended to.


On 12 August 2014 17:44, Antonio Augusto Santos  wrote:

> This one is for the devs, and Rashid in special: there is any new version
> of Kibana in the works?
> I'm asking this because I'm about to start a project in my company for log
> management, and there are some requisites to it (user separation, event
> correlation, histogram to compare two values, and so on).
>
> So, any changes of these functionalities landing on Kibana 4.0? ;)
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/2780b7a3-6312-4883-8a02-42b5aeefd88d%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAMH6%2Baw9%3DntGtezVXuzwftYDD7r8Q3r3yq1AwJ2mBY%3DsCMoktg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: About elasticsearch.org

[James Green]

The BGP routing table burst past 512k entries yesterday, I'm told. Any
"old" routers with that as a limit would have been affected limiting
traffic apparently at random. A few ISPs were caught out.


On 12 August 2014 18:21, Jack Park  wrote:

> I just hear second hand that the outage is pretty large.
>
> On Tue, Aug 12, 2014 at 10:14 AM, Jack Park 
> wrote:
> > It appears that liquidweb (their host) has some problems.  I and a
> > friend in Canada can open it on my cell phone, but nobody I know
> > around here can raise it on some other networks.
> >
> > On Tue, Aug 12, 2014 at 9:58 AM, Antonio Augusto Santos
> >  wrote:
> >> Working fine for me (in Brazil).
> >>
> >>
> >> On Tuesday, August 12, 2014 1:57:48 PM UTC-3, Jack Park wrote:
> >>>
> >>> Just curious:
> >>> on all browsers here in silicon valley, I cannot raise any
> >>> elasticsearch.org
> >>>
> >>> Is it just me (or comcast?)
> >>> Other websites appear fine.
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> Groups
> >> "elasticsearch" group.
> >> To unsubscribe from this group and stop receiving emails from it, send
> an
> >> email to elasticsearch+unsubscr...@googlegroups.com.
> >> To view this discussion on the web visit
> >>
> https://groups.google.com/d/msgid/elasticsearch/ee9afb15-2e69-400c-a21c-2017d3d818ea%40googlegroups.com
> .
> >> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAH6s0fwTcaAuAjBrB_8GnoBwxijBLvhbGOwuKV%2B37_6Tq958rw%40mail.gmail.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAMH6%2BazkwoeK7byAd98MA8iqOPmBTev_%2BhUuYAdEWzdiYWLAkA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: EsRejectedExecutionException: rejected execution (queue capacity 1000)

[James]

On Sat, 2014-08-09 at 23:53 -0700, Deep wrote:
> Hi,
> 
> 
> 
> Elastic search internally has thread pool and a queue size is
> associated with each pool. You can have pools for search threads,
> index threads etc. Please see the elastic search documentation in
> link 
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-threadpool.html
>  . I think it is possible to override these properties in the 
> elasticsearch.yml configuration file.
> 
> 
> Regards,
> Ishwardeep
> 
> On Saturday, 9 August 2014 00:54:02 UTC+5:30, digit...@gmail.com
> wrote:
> 
> So I've seen a few posts on this, but I've not seen any
> solutions posted.  I've been log monitoring and I was trying
> to determine how to fix the below...any information would be
> great thank you. 
> 
> [2014-08-08 19:14:12,578][DEBUG][action.search.type   ]
> [Jericho Drumm] [bro-201408032100][2],
> node[fgjxNK0cQ3O5Usn7wyjaMA], [P], s[STARTED]: Failed to
> execute
> [org.elasticsearch.action.search.SearchRequest@126067b7]
> lastShard [true]
> 
> org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: 
> rejected execution (queue capacity 1000) on 
> org.elasticsearch.search.action.SearchServiceTransportAction$23@5a879352
> at
> 
> org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62)
> at
> 
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
> at
> 
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
> at
> 
> org.elasticsearch.search.action.SearchServiceTransportAction.execute(SearchServiceTransportAction.java:509)
> at
> 
> org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:203)
> at
> 
> org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)
> at
> org.elasticsearch.action.search.type.TransportSearchTypeAction
> $BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:171)
> at
> org.elasticsearch.action.search.type.TransportSearchTypeAction
> $BaseAsyncAction.start(TransportSearchTypeAction.java:153)
> at
> 
> org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction.doExecute(TransportSearchQueryThenFetchAction.java:59)
> at
> 
> org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction.doExecute(TransportSearchQueryThenFetchAction.java:49)
> at
> 
> org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:63)
> at
> 
> org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:101)
> at
> 
> org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43)
> at
> 
> org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:63)
> at
> org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:92)
> at
> 
> org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:212)
> at
> 
> org.elasticsearch.rest.action.search.RestSearchAction.handleRequest(RestSearchAction.java:75)
> at
> 
> org.elasticsearch.rest.RestController.executeHandler(RestController.java:159)
> at
> 
> org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:142)
> at
> 
> org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:121)
> at org.elasticsearch.http.HttpServer
> $Dispatcher.dispatchRequest(HttpServer.java:83)
> at
> 
> org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:294)
> at
> 
> org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:44)
> at
> 
> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> at
> 
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline
> 
> $DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
> at
> 
> org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.jav

reduce-style aggregators

[James Cook]

There seems to be some reluctance by ES team to provide scrip table 
aggregators, or perhaps it's on a roadmap and just taking a long time. Kimchi 
has stated that he would like to identify these use cases and roll them into 
built-in aggregations so everyone can benefit. I think the range of these use 
cases is too broad for a specific set of implementations. The aggregations 
types included are fine for simple stats and bucketing, but there are probably 
hundreds of scenarios that require custom aggregation. 

I'll probably look into creating a custom aggregator for my use case. 

Note: a lot of the custom aggregators mentioned in this forum are general 
implementations of the reduce clause in a map reduce statement.

My use case is the implementation of an Item Response Theory algorithm on a 
filtered result set. The idea is that a list of student responses to a question 
can be incrementally processed to result in a proficiency value. The filter 
(map) results are restricted to a timeframe and sorted, and the aggregation 
(reduce) step will incrementally inject each students score into an algorithm 
that progressively converges on the student's ability magnitude to answer those 
questions. 

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4b8725e6-c73b-4da7-abeb-330c1e0d2406%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch completely unresponsive

[James]

On Thu, 2014-08-07 at 23:35 -0700, AsyncAwait wrote:

> What is your question?
> 


Oh..I was just wondering if there was any troubleshooting or debug steps
I could take.  Thanks for looking at this.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1407497371.2599.0.camel%40JamesiMac.
For more options, visit https://groups.google.com/d/optout.

has_parent slow in filter, fast in query?

[James Ekstrom]

We ran into a problem with a slow query.  Turns out having a has_child (or 
has_parent) query in the "filter" part of a "filtered" query is a lot 
slower than having the same has_child filter in the "query" part of the 
"filtered" query.

The faster of the two queries runs in under 100ms, while the slower one 
takes 250-300ms.  I don't understand why these would have different run 
times, any ideas?

The index is around 2GB. The child with about 600,000 docs, the parent with 
about 3,000,000 docs.

These are the two queries:

Slower:

"query": {
"filtered": {
  "filter": {
"and": {
  "filters": [
{
  "range": {
"price": {
  "gt": "0"
}
  }
},
{
  "has_parent": {
"type": "product",
"query": {
  "match_all": {}
}
  }
}
  ]
}
  }
}
  }

Faster:

"query": {
"filtered": {
  "query": {
"has_parent": {
  "type": "product",
  "query": {
"match_all": {}
  }
}
  },
  "filter": {
"and": {
  "filters": [
{
  "range": {
"price": {
  "gt": "0"
}
  }
}
  ]
}
  }
}
  }

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d811bca1-9798-49c5-b1f6-79cb5857f547%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Issues with index routing allocation in 1.3.1

[James Dooley]

I noticed an issue after upgrading from 0.90.9 to 1.3.1 that the 
index.routing.allocation.include._ip setting per index is not being 
properly handled.
I have used this setting in the past to move indices to an arching server 
for long term storage (outside of the cluster).

In  the past the past I have been able to do the following:

Pre-move settings:
  'index' => {
   'number_of_replicas' => '2',
   'version' => {
  'created' => '900999'
},
   'number_of_shards' => '4',
   'uuid' => 'xt8LJlsbR6COhx612BhiWg',
   'routing' => {
  'allocation' => {
'include' => {
   'tag' 
=> 'storage',
 },

'total_shards_per_node' => '2'
  }
}
 }
};



Move settings:
  'index' => {
   'number_of_replicas' => '2',
   'version' => {
  'created' => '900999'
},
   'number_of_shards' => '4',
   'uuid' => 'xt8LJlsbR6COhx612BhiWg',
   'routing' => {
  'allocation' => {
'include' => {
   'tag' 
=> '',
   '_ip' 
=> '192.168.0.51',
 },

'total_shards_per_node' => '999'
  }
}
 }
};



This would force one copy of each shard to migrate to the archiving server 
from the storage cluster.

Now in 1.3.1 the shards do not move at all with these settings, instead I 
have to set the following (which is incorrect):
  'index' => {
   'number_of_replicas' => '2',
   'version' => {
  'created' => '900999'
},
   'number_of_shards' => '4',
   'uuid' => 'xt8LJlsbR6COhx612BhiWg',
   'routing' => {
  'allocation' => {
'include' => {
   'tag' 
=> 'storage,backup',
   '_ip' 
=> '192.168.0.51',
 },

'total_shards_per_node' => '999'
  }
}
 }
};


According to the documentation "This is in contrast to include which will 
include a node if ANY rule matches."

I did a quick test and it does look like if I do not provide the _ip 
setting, but only use the tags setting the migration works as intended.  
This leads me to believe that the _ip field is being completely ignored.
In my use case I can't rely on the backup tag since I will have more than 
one backup server (using that tag), but need all of the data to go to a 
single server (ie need _ip, _host etc).

Unfortunately, I was running on an old cluster for awhile so I am not sure  
exactly where this broke at. But would have been somewhere between 0.90.9 
and 1.3.1.
Has anyone else run in to this issue?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f3a6af2c-7632-4329-8806-0dd0ae589068%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Disabling key_as_string

[James Griffin]

Cracking, thanks Colin

On Friday, August 1, 2014 3:28:45 PM UTC+1, Colin Goodheart-Smithe wrote:
>
> Sorry, I assumed you were using the histogram aggregation.  It was this 
> aggregation that the fixed was applied to. I have opened issue #7125 
> <https://github.com/elasticsearch/elasticsearch/issues/7125> to resolve 
> this in the terms aggregation.
>
> On Friday, 1 August 2014 14:04:52 UTC+1, James Griffin wrote:
>>
>> Hi Colin, 
>>
>> Thanks for getting back to me so quick. I'm running my query against a 
>> 1.3.1 cluster and still seeing "key_as_string", even though no format 
>> parameter is in the query. My query is below:
>>
>> {
>>   "query": {
>> "filtered": {
>>   "query": {
>> "match_all": {}
>>   }
>> }
>>   },
>>   "aggs": {
>> "topics": {
>>   "terms": {
>> "size": 10,
>> "field": "content.topics"
>>   }
>> }
>>   }
>> }
>>
>> In the mapping, the content.topics field is defined as an integer. The 
>> data was indexed pre-upgrade, though I can't imagine that would make a 
>> difference? Any ideas?
>>
>> Thanks, 
>> James
>>
>>
>> On Friday, August 1, 2014 12:40:24 PM UTC+1, Colin Goodheart-Smithe wrote:
>>>
>>> Hi James,
>>>
>>> This issue was brought up recently in the github issues list[1] and has 
>>> been fixed[2].  It should be available in the version 1.3.0.
>>>
>>> [1] https://github.com/elasticsearch/elasticsearch/issues/6655
>>>
>>> [2] https://github.com/elasticsearch/elasticsearch/pull/6830
>>>
>>> Regards,
>>>
>>> Colin
>>>
>>> On Friday, 1 August 2014 12:22:21 UTC+1, James Griffin wrote:
>>>>
>>>> Hi, 
>>>>
>>>> We're performing some aggregations with size=0 on a lot of data, and 
>>>> I'm looking to optimise the transfer over the wire. Our responses look 
>>>> like 
>>>> this:
>>>>
>>>> "buckets": [
>>>>
>>>>- {
>>>>   - "key": 15,
>>>>   - "key_as_string": "15",
>>>>   - "doc_count": 8183
>>>>},
>>>>- {
>>>>   - "key": 100,
>>>>   - "key_as_string": "100",
>>>>   - "doc_count": 4717
>>>>},
>>>>...
>>>>
>>>>
>>>> Obviously, the key_as_string field is adding nothing in this instance. 
>>>> Is there a way to ask for it to be excluded from the response?
>>>>
>>>> Cheers,
>>>> James
>>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/6a7bdf43-6658-4cfd-8b85-efc91cf9d21e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Disabling key_as_string

[James Griffin]

Hi, 

I dug into that PR and it seems it was solved for histograms but not for 
terms aggregations. I presume this is a new bug, then? Are there other 
aggregation types this should be solved for?

Thanks, 
James

On Friday, August 1, 2014 2:04:52 PM UTC+1, James Griffin wrote:
>
> Hi Colin, 
>
> Thanks for getting back to me so quick. I'm running my query against a 
> 1.3.1 cluster and still seeing "key_as_string", even though no format 
> parameter is in the query. My query is below:
>
> {
>   "query": {
> "filtered": {
>   "query": {
> "match_all": {}
>   }
> }
>   },
>   "aggs": {
> "topics": {
>   "terms": {
> "size": 10,
> "field": "content.topics"
>   }
> }
>   }
> }
>
> In the mapping, the content.topics field is defined as an integer. The 
> data was indexed pre-upgrade, though I can't imagine that would make a 
> difference? Any ideas?
>
> Thanks, 
> James
>
>
> On Friday, August 1, 2014 12:40:24 PM UTC+1, Colin Goodheart-Smithe wrote:
>>
>> Hi James,
>>
>> This issue was brought up recently in the github issues list[1] and has 
>> been fixed[2].  It should be available in the version 1.3.0.
>>
>> [1] https://github.com/elasticsearch/elasticsearch/issues/6655
>>
>> [2] https://github.com/elasticsearch/elasticsearch/pull/6830
>>
>> Regards,
>>
>> Colin
>>
>> On Friday, 1 August 2014 12:22:21 UTC+1, James Griffin wrote:
>>>
>>> Hi, 
>>>
>>> We're performing some aggregations with size=0 on a lot of data, and I'm 
>>> looking to optimise the transfer over the wire. Our responses look like 
>>> this:
>>>
>>> "buckets": [
>>>
>>>- {
>>>   - "key": 15,
>>>   - "key_as_string": "15",
>>>   - "doc_count": 8183
>>>},
>>>- {
>>>   - "key": 100,
>>>   - "key_as_string": "100",
>>>   - "doc_count": 4717
>>>},
>>>...
>>>
>>>
>>> Obviously, the key_as_string field is adding nothing in this instance. 
>>> Is there a way to ask for it to be excluded from the response?
>>>
>>> Cheers,
>>> James
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/15a97dd3-b3cf-46ea-a117-d0f7648b3c47%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Disabling key_as_string

[James Griffin]

Hi Colin, 

Thanks for getting back to me so quick. I'm running my query against a 
1.3.1 cluster and still seeing "key_as_string", even though no format 
parameter is in the query. My query is below:

{
  "query": {
"filtered": {
  "query": {
"match_all": {}
  }
}
  },
  "aggs": {
"topics": {
  "terms": {
"size": 10,
"field": "content.topics"
  }
}
  }
}

In the mapping, the content.topics field is defined as an integer. The data 
was indexed pre-upgrade, though I can't imagine that would make a 
difference? Any ideas?

Thanks, 
James


On Friday, August 1, 2014 12:40:24 PM UTC+1, Colin Goodheart-Smithe wrote:
>
> Hi James,
>
> This issue was brought up recently in the github issues list[1] and has 
> been fixed[2].  It should be available in the version 1.3.0.
>
> [1] https://github.com/elasticsearch/elasticsearch/issues/6655
>
> [2] https://github.com/elasticsearch/elasticsearch/pull/6830
>
> Regards,
>
> Colin
>
> On Friday, 1 August 2014 12:22:21 UTC+1, James Griffin wrote:
>>
>> Hi, 
>>
>> We're performing some aggregations with size=0 on a lot of data, and I'm 
>> looking to optimise the transfer over the wire. Our responses look like 
>> this:
>>
>> "buckets": [
>>
>>- {
>>   - "key": 15,
>>   - "key_as_string": "15",
>>   - "doc_count": 8183
>>},
>>- {
>>   - "key": 100,
>>   - "key_as_string": "100",
>>   - "doc_count": 4717
>>},
>>...
>>
>>
>> Obviously, the key_as_string field is adding nothing in this instance. Is 
>> there a way to ask for it to be excluded from the response?
>>
>> Cheers,
>> James
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1159e4b9-4278-4187-8f78-d404bc73b016%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Disabling key_as_string

[James Griffin]

Hi, 

We're performing some aggregations with size=0 on a lot of data, and I'm 
looking to optimise the transfer over the wire. Our responses look like 
this:

"buckets": [
   
   - {
  - "key": 15,
  - "key_as_string": "15",
  - "doc_count": 8183
   },
   - {
  - "key": 100,
  - "key_as_string": "100",
  - "doc_count": 4717
   },
   ...


Obviously, the key_as_string field is adding nothing in this instance. Is 
there a way to ask for it to be excluded from the response?

Cheers,
James

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b13ffb0f-9698-4154-bdcf-5b48da79c0e5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Pull Requests mounting?

[James Green]

Thanks Clint, my alarm was based on the lack of pulse activity over at
Kibana.


On 1 August 2014 09:37, Clinton Gormley 
wrote:

> Hi James
>
>
>
>> A cursory glance at the GitHub pages shows a build-up of pull requests:
>>
>> ElasticSearch: 135
>> Kibana: 64
>>
>> Is there a movement to get these merged or otherwise cleaned up? We are
>> waiting on at least one of these but lack hope in the face of the volumes
>> pending.
>>
>>
>>
> We have a concerted ongoing effort to work through all of these issues and
> PRs.  You can see from our Pulse page how much progress we're making:
>
> https://github.com/elasticsearch/elasticsearch/pulse/monthly
>
> Clint
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/ac9706f6-529c-4fe9-b52b-79f3420a0a29%40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/ac9706f6-529c-4fe9-b52b-79f3420a0a29%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAMH6%2Baz5JoHU2%3DrMk3151Vdr7wafktuBHUOkvG31uAcbgL_vJw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Pull Requests mounting?

[James Green]

Hi,

A cursory glance at the GitHub pages shows a build-up of pull requests:

ElasticSearch: 135
Kibana: 64

Is there a movement to get these merged or otherwise cleaned up? We are
waiting on at least one of these but lack hope in the face of the volumes
pending.

Thanks,

James

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAMH6%2Bawm3vRPA1NcyvWLTi-th3OnwzvjTRnC6rt9c%2B0CbSyHuw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] Elasticsearch Twitter River plugin 2.2.0 released

[James Green]

I was told a week or two ago on IRC that rivers were deprecated in favour
of external data sources like Logstash.

Is this not correct?


On 23 July 2014 13:00, Elasticsearch Team  wrote:

>  Heya,
>
> We are pleased to announce the release of the *Elasticsearch Twitter
> River plugin*, *version 2.2.0*
>
> The Twitter River plugin allows index twitter stream using elasticsearch
> rivers feature.
>
> Release Notes - Version 2.2.0 Fix
>
>- [62
>]
>- Generate default mapping even if index already exists
>- [60
>]
>- ignore_retweets does not ignore RT
>
> Update
>
>- [61
>]
>- Update to Twitter4J 4.0.2
>- [58
>]
>- Remove old deprecated user and password properties
>- [57
>]
>- Deprecate usage of camelCase settings
>- [54
>]
>- Added not_analyzed to retweet user_screen_name
>- [52
>]
>- Update to elasticsearch 1.2.0
>
> New
>
>- [56
>]
>- Add oauth credentials in `elasticsearch.yml` file
>- [51
>]
>- Support geo location array format
>- [50
>]
>- Add user stream support
>- [37
>]
>- Move tests to elasticsearch test framework
>
> Doc
>
>- [49
>]
>- [DOC] Link to rivers documentation is incorrect
>
> Issues, Pull requests, Feature requests are warmly welcome on
> elasticsearch-river-twitter
>  project
> repository!
>
> For questions or comments around this plugin, feel free to use
> elasticsearch mailing list
> !
>
> Enjoy,
>
> - The Elasticsearch team 
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/53cfa3ef.8124b40a.2b81.3571SMTPIN_ADDED_MISSING%40gmr-mx.google.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAMH6%2Bawb9LYHPnm_5yxVJBr6c6xNkuz200mjES52uyCOPTfK%3Dg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Solaris 10 mlockall error code

[James Pace]

Thanks for the information, I'll chase it up and let you know how I get on.

James

On Monday, 21 July 2014 12:36:09 UTC+1, Jörg Prante wrote:
>
> Error 11 is a POSIX error number (errno) and means EAGAIN, "Resource 
> temporarily unavailable", which is documented.
>
> On Solaris 10, this means, you must first allow the Elasticsearch user to 
> allocate this amount of virtual memory.
>
> Switch to Elasticsearch user and then check the following values
>
> prctl -n project.max-shm-memory $$
> prctl -n process.max-address-space $$
>
> Then the sys admin could create a project with projadd and projmod and 
> change resource limits for the Elasticsearch user.
>
> The error can also mean that Solaris has not enough memory for mlockall 
> because there is already software running using the memory, or it the free 
> memory is too fragmented.
>
> Jörg
>
>
>
> On Mon, Jul 21, 2014 at 11:09 AM, James Pace  > wrote:
>
>> We have been having issues running ES with the bootstrap.mlockall: true 
>> setting. We get the following error:
>>
>> [2014-07-21 09:56:44,436][WARN ][common.jna] Unknown mlockall error 11
>>
>> I have googled around and looked in the solaris documentation for the 
>> description of the error codes and I have been unsuccessful. The solaris 
>> docs are here 
>> <http://docs.oracle.com/cd/E26505_01/html/816-5168/mlockall-3c.html#REFMAN3Amlockall-3c>
>>  but 
>> they only list 3 error codes. Is the error code 11 generated by ES?
>>
>> Our box has a total of 64 gigs of RAM and we give 32gigs to ES. We are 
>> running it using Oracle Java 1.7.13 64bit. Any help on the matter would be 
>> greatly appreciated!
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/bc9ba19f-c798-4e7f-9977-c9707709c47e%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/elasticsearch/bc9ba19f-c798-4e7f-9977-c9707709c47e%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f503e0b0-2e6b-4b6d-9e0e-07c2858783df%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Solaris 10 mlockall error code

[James Pace]

Ooops good point! We are running version 1.2.1

On Monday, 21 July 2014 10:18:15 UTC+1, Mark Walkom wrote:
>
> What elasticsearch version are you on?
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com 
> web: www.campaignmonitor.com
>
>
> On 21 July 2014 19:09, James Pace > 
> wrote:
>
>> We have been having issues running ES with the bootstrap.mlockall: true 
>> setting. We get the following error:
>>
>> [2014-07-21 09:56:44,436][WARN ][common.jna] Unknown mlockall error 11
>>
>> I have googled around and looked in the solaris documentation for the 
>> description of the error codes and I have been unsuccessful. The solaris 
>> docs are here 
>> <http://docs.oracle.com/cd/E26505_01/html/816-5168/mlockall-3c.html#REFMAN3Amlockall-3c>
>>  but 
>> they only list 3 error codes. Is the error code 11 generated by ES?
>>
>> Our box has a total of 64 gigs of RAM and we give 32gigs to ES. We are 
>> running it using Oracle Java 1.7.13 64bit. Any help on the matter would be 
>> greatly appreciated!
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/bc9ba19f-c798-4e7f-9977-c9707709c47e%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/elasticsearch/bc9ba19f-c798-4e7f-9977-c9707709c47e%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/9b4fa7b1-2561-4f33-8a60-969ff317206b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Solaris 10 mlockall error code

[James Pace]

We have been having issues running ES with the bootstrap.mlockall: true 
setting. We get the following error:

[2014-07-21 09:56:44,436][WARN ][common.jna] Unknown mlockall error 11

I have googled around and looked in the solaris documentation for the 
description of the error codes and I have been unsuccessful. The solaris 
docs are here 

 but 
they only list 3 error codes. Is the error code 11 generated by ES?

Our box has a total of 64 gigs of RAM and we give 32gigs to ES. We are 
running it using Oracle Java 1.7.13 64bit. Any help on the matter would be 
greatly appreciated!

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/bc9ba19f-c798-4e7f-9977-c9707709c47e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

ElasticSearch Hadoop

[James Cook]

I've read through much of the documentation for es-hadoop, but I might be 
coming away with some misunderstandings.

The setup docs for elasticsearch for apache hadoop (es-hadoop) uses the 
word *interact* which is a bit vague.

Elasticsearch for Apache Hadoop is an open-source, stand-alone, 
> self-contained, small library that allows Hadoop jobs (whether using 
> Map/Reduce or libraries built upon it such as Hive, Pig or Cascading) to 
> interact with Elasticsearch. Data flows bi-directionaly so that 
> applications can leverage transparently the Elasticsearch engine 
> capabilities to significantly enrich their capabilities and increase the 
> performance.


So, does this mean I have a separate Hadoop instance (potentially built 
upon HDFS or AWS EMR) and I can query data using either the elasticsearch 
(REST/Java/etc) or hadoop (Hive, Pig, Cascading) environments?



 

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/c12c35a0-b9ee-461e-8e81-12910dd06894%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

REST vs URL searching - what's the difference in this query?

[James Green]

This works in a browser and finds tens of thousands of records:

http://localhost:9200/accountevents/_search?q=Medium:SMS

This cmd finds nothing:

curl -XGET 'http://localhost:9200/accountevents/_search' -d '{
  "query": {
"term": {
  "Medium": "SMS"
  }
  }
}'

We can't work out what the difference is. Can someone point it out to us?

Thanks,

James

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d4b85b87-8b79-4faa-82f1-d4706f1616e5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.