Re: Exact duplicate results (same _id) for a search query. Is this a bug?
Thank you Alex, Jörg & Itamar. I am using routing, so a mistake in routing looks like the most likely culprit. The two documents are on different shards. To answer the other questions: single server, optimize does not alter it, and I believe it's been version 1.2.1 from setup. Best regards, - Daniel -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEmLStmmmdu2kOWMhvpQgfU4Zq_x6B55ZoGO3Oamh71VvR5ybA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Exact duplicate results (same _id) for a search query. Is this a bug?
Hello, I have an elasticsearch index which is returning two identical results. I don't mean 2 copies of a similar document. These results have the same elasticsearch _id. Details below. Does anyone know why this happens? Is it a bug? Best regards, - Daniel Version: 1.2.1 Query: http://localhost:9200/workspace/group/_search?q=winterwell Result: { - took: 8, - timed_out: false, - _shards: { - total: 5, - successful: 5, - failed: 0 }, - hits: { - total: 2, - max_score: 0.89743817, - hits: [ - { - _index: "workspace", - _type: "group", - _id: "winterwell@DBGroup", - _score: 0.89743817, - _source: { - name: "winterwell", - tags: { }, - ...some details skipped... } }, - { - _index: "workspace", - _type: "group", - _id: "winterwell@DBGroup", - _score: 0.89743817, - _source: { - name: "winterwell", - tags: { }, - ...some details skipped... } } ] } } -- Dr Daniel Winterstein Director A: CodeBase Argyle House, Edinburgh, EH3 9DR M: +44 (0)772 5172 612 http://winterwell.com http://sodash.com -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c19b5667-12af-4d4c-8457-05361a926d66%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Could a custom Aggregator be used for general purpose Map/Reduce or bulk update?
Hello, So by writing a plugin you can create a custom aggregation.[1] I'd like to explore what we could do with that. Why? I'm looking for ways round a costly scan-and-update-each-document algorithm. Do Aggregators run in a parallel fashion, with your aggregation being run against all shards at once? Or do they go through the shards sequentially? How does an Aggregator run only-once for each matching document? I.e. if we have a shard replicated on 3 nodes... Does the aggregation pick one node for that shard? Or does it build up in memory a set of seen documents to avoid duplicating? What happens if you make calls to ElasticSearch from within an Aggregator? Such as updating a document. What about updating the context document that the Aggregator is looking at then-and-there -- could that be done efficiently from within the Aggregator? If so, would you & could you override something in the Aggregator class so it runs over every copy of every document? Thank you for any help with these questions! Best regards, - Daniel 1: https://groups.google.com/forum/#!searchin/elasticsearch/aggregations/elasticsearch/0UYLbyeWiw4/RlSnJtgDj0AJ -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a635a37d-7acd-4ca3-af00-dea882ed27ae%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Dynamic mapping with *lots* of fields?
Hello Jörg, > What is your data model? > It seems you could live with search on a single field only. A relevant point I left out: I have concurrent non-conflicting edits going on, which I want to merge. I.e. several users setting their own dynamic fields at the same time. Using separate fields with the update command makes this painless. NB: I'm assuming ES handles concurrent updates nicely, but I haven't stress-tested this yet. If we had a single-field, we'd need to introduce versioning and conflict handling. > Then I conclude why not use key/value stream, search on value, and use highlighting for looking up the key(s) in the result doc? I'm not sure I understand what you're proposing here. NB: The alternatives I'm considering are: 1. Lots of dynamic fields -- with the concerns talked about in this thread. 2. In my code, splitting edits to the "concurrent fields" off from other edits. Then using normal put and update for normal fields, and MVEL update scripts for edits to concurrent fields. 3. Versioning and conflict-handling. 4. Writing a plugin for the ES backend to allow for custom merge logic on a single field. Best regards, - Daniel On Monday, 28 April 2014 08:13:17 UTC+1, Jörg Prante wrote: > > What is your data model? > > It seems you could live with search on a single field only. > > Then I conclude why not use key/value stream, search on value, and use > highlighting for looking up the key(s) in the result doc? > > { >"key" : "...", >"value" : "..." > } > > Jörg > > > On Sun, Apr 27, 2014 at 1:15 PM, Daniel Winterstein < > daniel.wi...@gmail.com > wrote: > >> Hello, >> >> I have a case where each document is likely to introduce some specific >> & often unique properties -- leading to millions or more fields. >> The values stored under these should be searchable. But the mapping >> structure itself is not used in queries. >> >> Having ElasticSearch treat these as dynamic fields works -- but how >> well does ElasticSearch cope with very large mappings? >> >> Am I right in thinking each property adds a field to the mapping, >> which is shared & held in memory by all nodes? >> So a mapping can become too large, and create performance issues. E.g >> 100 million documents might lead to gigabytes of ram locked up >> modelling the mapping. >> Is that correct? >> >> You can switch off dynamic fields altogether -- but then I think the >> content stored within them becomes unsearchable. >> Is there a way to have dynamic fields accessible to search, without >> modelling them as part of the mapping? >> >> Thank you & >> Best regards, >> - Daniel >> >> -- >> Dr Daniel Winterstein >> Director >> >> A: CodeBase Argyle House, Edinburgh, EH3 9DR >> M: +44 (0)772 5172 612 >> http://winterwell.com http://sodash.com >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elasticsearc...@googlegroups.com . >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAEmLStmrGs%2BHaW%3De1XChi%3DFQ_LKEPEntRUvT8zt_BJKPUESkeg%40mail.gmail.com >> . >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0f402257-4d5a-4b69-b0a3-4cc5e939aaa8%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Dynamic mapping with *lots* of fields?
Hello, I have a case where each document is likely to introduce some specific & often unique properties -- leading to millions or more fields. The values stored under these should be searchable. But the mapping structure itself is not used in queries. Having ElasticSearch treat these as dynamic fields works -- but how well does ElasticSearch cope with very large mappings? Am I right in thinking each property adds a field to the mapping, which is shared & held in memory by all nodes? So a mapping can become too large, and create performance issues. E.g 100 million documents might lead to gigabytes of ram locked up modelling the mapping. Is that correct? You can switch off dynamic fields altogether -- but then I think the content stored within them becomes unsearchable. Is there a way to have dynamic fields accessible to search, without modelling them as part of the mapping? Thank you & Best regards, - Daniel -- Dr Daniel Winterstein Director A: CodeBase Argyle House, Edinburgh, EH3 9DR M: +44 (0)772 5172 612 http://winterwell.com http://sodash.com -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEmLStmrGs%2BHaW%3De1XChi%3DFQ_LKEPEntRUvT8zt_BJKPUESkeg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Precise shard routing, remote document specific nodes
Hello, I have a similar requirement for some documents to be stored only on the local node. A local index as @Kimchy suggests sounds like a good solution -- but how do you create one? Thank you, - Daniel -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d622db36-1979-4662-8c6a-847040a87207%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Put mapping documentation -- What options are available? Specifically, how to store a property but without indexing it?
Sorry Ivan! I'm not having much luck on this thread. Daniel Sent from my phone. Please excuse the brevity. On 26 Feb 2014 01:58, "Ivan Brusic" wrote: > Luke? :) > > > On Tue, Feb 25, 2014 at 1:09 PM, Daniel Winterstein < > daniel.winterst...@gmail.com> wrote: > >> Dear Hariharan, Alex, Luke, >> >> My apologies. You're quite right. The information is there -- I just >> didn't read far enough down. >> >> Thank you for your help & persistence. >> >> Best regards, >> - Daniel >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elasticsearch+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAEmLStnHQCUuMPJHhbcoq8_iQgFX%3D22t9%3DS9gOwWC7C1OtDToA%40mail.gmail.com >> . >> For more options, visit https://groups.google.com/groups/opt_out. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/elasticsearch/qER5uOq2A20/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQD%3Dk0htmXcEwXBBB4T%2BwqNAyA_fOz41DX5cinf3aYsQGg%40mail.gmail.com > . > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEmLSt%3DjVdN_j1GQ3BYbpW0REeTOSLOJdH39UxCTmy7mnsbpEg%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Put mapping documentation -- What options are available? Specifically, how to store a property but without indexing it?
Dear Hariharan, Alex, Luke, My apologies. You're quite right. The information is there -- I just didn't read far enough down. Thank you for your help & persistence. Best regards, - Daniel -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEmLStnHQCUuMPJHhbcoq8_iQgFX%3D22t9%3DS9gOwWC7C1OtDToA%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Put mapping documentation -- What options are available? Specifically, how to store a property but without indexing it?
Hello Alex, To take the example from that page: $ curl -XPUT 'http://localhost:9200/twitter/tweet/_mapping' -d ' { "tweet" : { "properties" : { "message" : {"type" : "string", // What can go here?? // I've seen "analyzer", "store", "enabled" used in passing in examples without explanation. // Somewhere there must be a list of these! What they mean, and what the supported values are. } } } } ' > Specifically, I have some large data properties I want to store and retrieve > with results, but not have them indexed for search. So help with that would > be great. I think "enabled" might fit this. Best regards, - Daniel -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEmLStmw9DtvyWLbyHbmmrmqc8PoZArWt1aCH637NGyyUqqFuw%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Put mapping documentation -- What options are available? Specifically, how to store a property but without indexing it?
Thank you Hariharan, but no. I am looking for documentation for the other options, such as analyze and store. Best regards, Daniel Sent from my phone. Please excuse the brevity. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEmLSt%3D_vb4x7zYor6gH%3DWi%2BAir3Kgf7NoV0kgzUy4N2RfFpTQ%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Put mapping documentation -- What options are available? Specifically, how to store a property but without indexing it?
Hello, The documentation at http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-put-mapping.html is a little sparse on details. Where can I find documentation for the options/parameters that put mapping takes? E.g. store, analyze, and just a list of what other options there are. Specifically, I have some large data properties I want to store and retrieve with results, but not have them indexed for search. So help with that would be great. Thank you & best regards, - Daniel -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/dfee8225-809b-4e7f-a2bc-3a032b49c4ab%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Which node(s) hold a document?
Thanks Luca -- that should do the job. The cat shards endpoint also looks relevant: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cat-shards.html Best regards, - Daniel On 17 February 2014 08:43, Luca Cavanna wrote: > Hi, > I'd have a look at the search shards api, which given a search request > returns which shards (and nodes) it would get executed on, without actually > executing it. Have a look at issue #2726 . > > > On Sunday, February 16, 2014 11:09:01 PM UTC+1, Daniel Winterstein wrote: >> >> Hello, >> >> Is it possible to find out (a) which logical shard a document is in, and >> (b) which servers are holding that shard? >> These are documents with parents, so setRouting() is used when indexing. >> >> NB: I'm working in Java using the Java client. >> >> Why? >> I want to do distributed computation, where the processing for a document >> happens on the server holding the data. >> I'm using a parent-child routing scheme, which means that large blocks of >> related data will end up in the same shard. >> Shifting computation to be near data, rather than vice-versa, would make a >> significant difference. >> Also, I'm thinking the holding servers can be used as part of coordinating >> the distribution of jobs. >> >> Thank you for any help, >> - Daniel >> > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/elasticsearch/XEJECqcdJhU/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/42962441-8178-48b2-92c3-4f92da1721f2%40googlegroups.com. > > For more options, visit https://groups.google.com/groups/opt_out. -- Dr Daniel Winterstein Director A: TechCube, Edinburgh, EH9 1PL M: +44 (0)772 5172 612 http://winterwell.com http://sodash.com -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEmLStmH6-zvbHvhkE_nuBLRUc7yoyn0SpDUCyFtcd0OW-eivw%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Which node(s) hold a document?
Hello, Is it possible to find out (a) which logical shard a document is in, and (b) which servers are holding that shard? These are documents with parents, so setRouting() is used when indexing. NB: I'm working in Java using the Java client. Why? I want to do distributed computation, where the processing for a document happens on the server holding the data. I'm using a parent-child routing scheme, which means that large blocks of related data will end up in the same shard. Shifting computation to be near data, rather than vice-versa, would make a significant difference. Also, I'm thinking the holding servers can be used as part of coordinating the distribution of jobs. Thank you for any help, - Daniel -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6ade77fb-3752-4db6-b678-e23c7118de08%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Returning values from an update script
Thanks Alexander. fields might be exactly what I need. Do you know what happens if there is a race / conflict-and-retry situation? Will the field values returned for a request always be the ones set by that request? Or can you do an update, but get a returned value which comes from a different but concurrent update? Thank you for your time & help, - Daniel On 27 January 2014 10:17, Alexander Reelsen wrote: > Hey, > > you can return fields from the update response (even the whole source), > see the end of > http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-update.html > You could then compare on the application side, if something had changed. > Does that help? > > > --Alex > > > > On Fri, Jan 24, 2014 at 1:40 AM, Daniel Winterstein < > daniel.winterst...@gmail.com> wrote: > >> Hello, >> >> I'm running an update request using a script. How can I return values >> from that? >> >> Specifically, I want to return true/false based on what update the script >> has done. >> >> Thank you & >> Best regards, >> - Daniel >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elasticsearch+unsubscr...@googlegroups.com. >> >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/cffb6275-cb99-444a-acbf-50e8ed724990%40googlegroups.com >> . >> For more options, visit https://groups.google.com/groups/opt_out. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/elasticsearch/JeOQUtO1G1E/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAGCwEM_xJCWFwA25cs8ktforfMDh9khx22mNArPFr6xxJ4fs0w%40mail.gmail.com > . > > For more options, visit https://groups.google.com/groups/opt_out. > -- Dr Daniel Winterstein Director A: TechCube, Edinburgh, EH9 1PL M: +44 (0)772 5172 612 http://winterwell.com http://sodash.com -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEmLSt%3DJq%3D0vFhrKoR8b1qnVx2eaiXhqMAdy3s29dD65W%3D94Mg%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Bug Report: Search on elasticsearch.org itself can break
Update: Someone has partly fixed this, but it's still buggy. The search "ctx" (http://www.elasticsearch.org/?s=ctx) now returns a list of results. But the results 404, e.g. The result "update » 1.0 API » js" links to http://www.elasticsearch.org/guide/en/elasticsearch/client/javascript-api/current/api-reference/#api-update which doesn't exist. Best regards, - Daniel On 24 January 2014 12:04, Daniel Winterstein wrote: > Hello, > > I don't know what the process for reporting bugs is, but I figure someone at > elasticsearch.org should know their site-search has one. > > The following url reliably generates an error: > http://www.elasticsearch.org/?s=ctx > > search results for "ctx" > > Warning: Invalid argument supplied for foreach() > in/home/elastic/deploys/production.elasticsearch.org/releases/20130624161346/org.elasticsearch.www/content/themes/elasticsearch-org/search.php > on line 36 > For future reference: what is the best way to report bugs? Via github?Best > regards, - Daniel > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/elasticsearch/97l7MkWfymU/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/7aa3d47e-2c43-4850-b7a6-75d34106c94d%40googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. -- Dr Daniel Winterstein Director A: TechCube, Edinburgh, EH9 1PL M: +44 (0)772 5172 612 http://winterwell.com http://sodash.com -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEmLStmo28GqXODeSBOObBzAQzO%2BrWE24%3DY%3Da8dxfjj1cpbnHg%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Bug Report: Search on elasticsearch.org itself can break
Hello, I don't know what the process for reporting bugs is, but I figure someone at elasticsearch.org should know their site-search has one. The following url reliably generates an error: http://www.elasticsearch.org/?s=ctx search results for "ctx"*Warning*: Invalid argument supplied for foreach() in */home/elastic/deploys/production.elasticsearch.org/releases/20130624161346/org.elasticsearch.www/content/themes/elasticsearch-org/search.php* on line *36* For future reference: what is the best way to report bugs? Via github?Best regards, - Daniel -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7aa3d47e-2c43-4850-b7a6-75d34106c94d%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Returning values from an update script
Hello, I'm running an update request using a script. How can I return values from that? Specifically, I want to return true/false based on what update the script has done. Thank you & Best regards, - Daniel -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/cffb6275-cb99-444a-acbf-50e8ed724990%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.