date:20140812

Why it would be expensive?

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 13 août 2014 à 03:43, Jianyi  a écrit :

Hi David, 

I'm afraid it would be a little expensive to move to ES.

I came across elephant-bird-lucene of twitter yesterday, and I will try it.

Thanks for your reply.



--
View this message in context: 
http://elasticsearch-users.115913.n3.nabble.com/what-s-the-difference-between-solr-and-elasticsearch-in-hdfs-case-tp4061659p4061760.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1407894195068-4061760.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/BAEF859C-F48F-410B-9D6F-AD7C26F6E35B%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Re: what's the difference between solr and elasticsearch in hdfs case?

Hi David, 

I'm afraid it would be a little expensive to move to ES.

I came across elephant-bird-lucene of twitter yesterday, and I will try it.

Thanks for your reply.



--
View this message in context: 
http://elasticsearch-users.115913.n3.nabble.com/what-s-the-difference-between-solr-and-elasticsearch-in-hdfs-case-tp4061659p4061760.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1407894195068-4061760.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.

Re: Can't delete Index!

Great, thanks for the help. I did few things and I am not sure which ended 
up wiping the index. I killed the cluster, I upgraded to 1.2.2, I rebooted 
the 2 AWS instances, I indexed on each node separately, then I joined the 
cluster. Things look fine now.

On Tuesday, August 12, 2014 10:46:33 AM UTC-4, Jörg Prante wrote:
>
> There is one node with zen unicast that can not connect. I do not know how 
> to find more about that, it seems an EC2 issue.
>
> Except you are being attacked in vain by the scripting vulnerability, 
> there are only syntax errors in configuration and queries... you should be 
> able to fix that.
>
> Jörg
>  
>
>
> On Tue, Aug 12, 2014 at 4:25 PM, Sam2014 
> > wrote:
>
>> Just wondering, will remove/re-install get rid of the stagnant index 
>> since I plan on upgrading from 1.2.1 to 1.2.2?
>>
>>
>> On Tuesday, August 12, 2014 9:42:10 AM UTC-4, Sam2014 wrote:
>>>
>>> The link doesn't work.
>>> I am not sure about the difference, I have always deleted the indices 
>>> the same way, curl -XPUT http... //to create, and curl -XDelete /http:...
>>> I did not create any aliases, I am just using Get _aliases to see whats 
>>> still on the node.
>>> Did the log tell you anything, there are some weird errors in there that 
>>> I don't know how to interpret, a quick look there may tell you something.
>>>
>>> On Tuesday, August 12, 2014 3:04:39 AM UTC-4, Jörg Prante wrote:

 So you are wrestling with aliases. You can not delete aliases by file 
 system operations. Have you checked

 http://www.elasticsearch.org/guide/en/elasticsearch/
 reference/current/indices-aliases.html#deleting

 for deleting aliases?

 Jörg


 On Tue, Aug 12, 2014 at 4:10 AM, Sam2014  wrote:

> ///Attached is the cluster log that covers the whole sequence from 
> shutting the cluster and re-activating it.
>
> //Show me current state of cluster
> {
>"cluster_name": "rexCluster",
>"status": "red",
>"timed_out": false,
>"number_of_nodes": 2,
>"number_of_data_nodes": 2,
>"active_primary_shards": 7,
>"active_shards": 7,
>"relocating_shards": 0,
>"initializing_shards": 2,
>"unassigned_shards": 21
> }
>
> //Show me indices on the cluster
>
> GET _aliases
> {
>"foo": {
>   "aliases": {}
>},
>"mediregs": {
>   "aliases": {}
>},
>"twitter": {
>   "aliases": {}
>}
> }
>
>
>
> //Shut down the cluster
>
> POST /_cluster/nodes/_shutdown
> {
>"cluster_name": "rexCluster",
>"nodes": {
>   "_vPEfmc9TruIpugk4NWleg": {
>  "name": "node 1"
>   },
>   "DB94rWiiSvS9HUTXwI7GTw": {
>  "name": "node 2"
>   }
>}
> }
>
>  //Go into the fs and delete files under /data
>
> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0$
>  
> ls
>
> *indices*  node.lock  *_state*
>
> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0$
>  
> cd indices/
>
> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$
>  
> ls
>
> *foo*  *mediregs*  *twitter*
>
> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$
>  
> sudo rm -rf foo
>
> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$
>  
> sudo rm -rf mediregs
>
> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$
>  
> sudo rm -rf twitter
>
> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$
>  
> ls
>
> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/
> rexCluster/nodes/0/indices$ 
>  //Restart nodes 1 &2 - should be in log file
>
> Get _aliases //Again - they came back!!
> {
>"foo": {
>   "aliases": {}
>},
>"mediregs": {
>   "aliases": {}
>},
>"twitter": {
>   "aliases": {}
>}
> }
>
>
> //Ok - Try to delete index
> DELETE /mediregs
>
> //takes little while and comes back
> {
>"acknowledged": false
> }
>
>
>
>
> On Monday, August 11, 2014 7:12:22 PM UTC-4, Jörg Prante wrote:
>
>> Can you post your detailed steps and debug logs?
>>
>> Make also sure there is no third node - this looks like ES is 
>> shipping shards to a third node from which the two nodes are recovering 
>> from.
>>
>> Jörg
>>
>>
>> On Tue, Aug 12, 2014 at 1:06 AM, Sam2014  wrote:
>>
>>> I stopped node 1, then I stopped node 2. I disabled the cluster, 
>>> deleted files from nodes, Made sure n

Re: ES service self update?

2014-08-12 Thread Blake Atkinson

If you haven't already, see: 
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html#rolling-upgrades

In general, I would advise against doing automated updating unless you have 
a significantly large cluster size. Mostly because nothing is guaranteed 
version to version and there might be caveats for new releases you might 
not be aware of before your cluster has already started the process. It 
would be very simple for something to go wrong that might affect data 
integrity and/or cluster stability.

If you must proceed down this route, create a small staging/test cluster w/ 
data and test the upgrade there first. Hopefully you're using VMs so you 
can snapshot and observe your upgrade's behavior before affecting 
production systems.

Note: If you were thinking of updating a node in-place, you should instead 
try symlinking the production ES service so you can then (1) download the 
new version, (2) stop shard allocation and ES, (3) update/change the 
symlink, and (4) restart ES. Also covered in the link above.

On Monday, August 11, 2014 1:39:24 PM UTC-5, kti...@hotmail.com wrote:
>
> I am very new to Elastic Search. I was curious if ES service can 
> periodically check for the latest version such as 1.1.3 and update nodes 
> one at a time silently?
> I am in the process of writing PowerShell script to do this but thought 
> ES might have solution for this already.
>
> thanks
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e7b0953a-6853-4ba7-a63d-858ed2c93be9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to receive part of the text field?

2014-08-12 Thread Nikolas Everett

I imagine script field can do this.
On Aug 12, 2014 6:38 PM, "Ivan Brusic"  wrote:

> If the 200kb number is fixed, then the simplest solution would be to store
> that content separately in a new field. It does not need to be analyzed,
> just stored.
>
> Perhaps highlighters might work. Never used them, so it is just a guess.
>
> Cheers,
>
> Ivan
> On Aug 12, 2014 8:17 AM, "Dmitriy Bashkalin"  wrote:
>
>> Hello.
>> There is a field with large amount of text. How I can get part of value,
>> such as 200KB? By analogy with the from () / size ().
>> Like as "Give me the first 200KB part of object with this Id"
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/bf664bcf-f936-4391-84d9-46cd26cefdd6%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAa54%2B5hiB1R4twZ3Xwkb34QL56jHDmCzET1NNtn%2B4now%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1TYg-SS%3Df9z%3DfpYoZMyDVjv_pcnOVrmRNFa9ufq8H%2BaA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to receive part of the text field?

2014-08-12 Thread Ivan Brusic

If the 200kb number is fixed, then the simplest solution would be to store
that content separately in a new field. It does not need to be analyzed,
just stored.

Perhaps highlighters might work. Never used them, so it is just a guess.

Cheers,

Ivan
On Aug 12, 2014 8:17 AM, "Dmitriy Bashkalin"  wrote:

> Hello.
> There is a field with large amount of text. How I can get part of value,
> such as 200KB? By analogy with the from () / size ().
> Like as "Give me the first 200KB part of object with this Id"
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/bf664bcf-f936-4391-84d9-46cd26cefdd6%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAa54%2B5hiB1R4twZ3Xwkb34QL56jHDmCzET1NNtn%2B4now%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Unallocated shards with empty nodes

2014-08-12 Thread Christopher Ambler

To add a little more information, the six nodes are broken up into three 
groups. The first two have node.zone: first, the second are node.zone: 
second and the third are node.zone: third.

I also have "cluster.routing.allocation.awareness.attributes: zone" in my 
config.

So as you can see, the empty nodes are the two that are "node.zone: second" 
in their config file.

Could this be involved?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/452f4419-265b-4e37-aaeb-094031914bc3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: impact of stored fields on performance

2014-08-12 Thread Ashish Mishra

The query size parameter is 200.  
Actual hit totals vary widely, generally around 1000-1.  A minority are 
much lower.  About 10% of queries end up with just 1 or 0 hits.


On Tuesday, August 12, 2014 6:31:29 AM UTC-7, Adrien Grand wrote:
>
> Hi Ashish,
>
> How many documents do your queries typically retrieve? (the value of the 
> `size` parameter)
>
>
> On Tue, Aug 12, 2014 at 12:48 AM, Ashish Mishra  > wrote:
>
>> I recently added a binary type field to all documents with mapping 
>> "store": "true".  The field contents are large and as a result the on-disk 
>> index size rose by 3x, from 2.5Gb/shard to ~8Gb/shard.  
>>
>> After this change I've seen a big jump in query latency.  Searches which 
>> previously took 40-60ms now take 800ms and longer.  This is the case even 
>> for queries which *don't* return the binary field.
>> I tried optimizing the index down to max_num_segments=1, but query 
>> latency remains high.
>>
>> Is this expected?  Obviously queries returning the new field will take a 
>> hit (since field data needs to be loaded from disk).  But I would've 
>> expected other queries should not be much affected.
>>
>> Is the problem that larger file sizes make memory-mapping and the FS 
>> cache less efficient?  Or are stored fields still getting loaded from disk 
>> even when not included in the "fields" term?
>>  
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/6ef50cab-3004-490b-bc2d-ea7e71a824a5%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> -- 
> Adrien Grand
>  

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1105d739-114e-4047-994e-aba8e27066b3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: A fresh variant of MapperParsingException

Thanks!
The top level of my mappings.json is changed from the gist to look like this:

{
"core": {
"properties": {
"lox": {

And that appears to work.

On Tue, Aug 12, 2014 at 8:43 AM, Jun Ohtani  wrote:
> hi,
>
> you should not remove "core".
> please try to add that.
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-put-mapping.html
>
> 2014年8月12日火曜日、Jack Parkさんは書きました:
>>
>> I created a gist for the mappings I am sending through node-es, with a
>> returned error:
>> MapperParsingException[Root type mapping not empty after parsing!
>>
>> The gist is at
>> https://gist.github.com/KnowledgeGarden/b965b7e78f19f9be9025
>>
>> Note, that if I remove the upper portion of the json:
>> "topics": {
>> "mappings": {
>> "core": {
>> and just send in from "properties":, the same error occurs.
>>
>> The exact code which sends mappings into node-es is found here:
>>
>> https://github.com/KnowledgeGarden/TQTopicMap/blob/master/lib/persist/es.js
>>
>> I'm wondering if anyone recognizes an error in my mapping. I
>> deliberately include the index and type in the mapping. It seems hard
>> to find examples of schema mappings that do what I am trying to do.
>>
>> Thanks!
>>
>> Here is the error message:
>> ESClient-1 Error: {"error":"MapperParsingException[mapping [topics]];
>> nested: Ma
>> pperParsingException[Root type mapping not empty after parsing! Remaining
>> fields
>> : [mappings : {core={properties={inOf={index=not_analyzed, store=true,
>> type=stri
>> ng}, _ver={index=not_analyzed, store=true, type=string},
>> tpC={index=not_analyzed
>> , store=true, type=long}, lox={index=not_analyzed, store=true,
>> type=string}, srt
>> Dt={index=not_analyzed, store=true, type=long}, url={index=not_analyzed,
>> store=t
>> rue, type=string}, lists={properties={body={index=analyzed, store=true,
>> type=str
>> ing}, sbOf={index=not_analyzed, store=true, type=string},
>> details={index=analyze
>> d, store=true, type=string}, psi={index=not_analyzed, store=true,
>> type=string},
>> label={index=analyzed, store=true, type=string}, subj={index=analyzed,
>> store=tru
>> e, type=string}}]]; ","status":400}
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CAH6s0fxkx%3D0ZpDM-qAEi4RREKKVubUcX%3DVSPSz5mwGTxYuWNNw%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
> ---
> Jun Ohtani
> blog : http://blog.johtani.info
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAPW8A5zBLQNyrwX0HZwc6CECYV6H%3DqHUS5aKuy%2BCouO6b%2B9EtQ%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAH6s0fzQvAoXbEbJ09Fdit6HF_nzJqZ7kRac9MAZ8rrqtHRE%2Bw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: github issues search query DSL

My recommendation is to use the simple_query_string, which is a mini
language for itself.

Beside this, I am about completing a plugin for Contextual Query Language
(CQL) and Search/Retrieve via URL (SRU) for bibliographic searches, which
is very close to your question.

https://github.com/xbib/elasticsearch-sru

The reason is I'm connecting legacy library systems to Elasticsearch for
very basic queries like book searches. In CQL, imagine queries like this

dc.creator = "Harriet Beecher Stowe" and dc.title = "Uncle Tom's cabin"

on docs like this

{
"dc" : {
"title" : "Uncle Tom's cabin",
"creator" : "Harriet Beecher Stowe"
}
}

With SRU 2.0/searchRetrieve 1.0, you can even add parameters for facets =
simple terms aggregation.

At the moment, I'm adding Handlebars templates for rendering ES
SearchResponse to SRU XML (which is part of the legacy I have to address)

Jörg



On Tue, Aug 12, 2014 at 7:39 PM, Ivan Brusic  wrote:

> Look into the Lucene query parser, which its the syntax that the query
> string query uses. After that, look into the various Lucene contrib modules
> that extend the query syntax (span near is one).
>
> I do not think that anyone has implemented a new query parser as an
> elasticsearch plugin yet, but I could be wrong.
>
> Cheers,
>
> Ivan
> On Aug 12, 2014 1:33 PM, "Erik Paulson"  wrote:
>
>> Hello -
>>
>> Github has a pretty slick search interface for issues, complete with a
>> set of qualifiers that users can stick onto their free-form text queries.
>> They're using ES for their code search, and I'm guessing for their issues
>> as well:
>>
>> https://help.github.com/articles/searching-issues
>>
>> Is there a good library or example code somewhere that implements a
>> similar query language? Obviously some qualifiers are github-specific, but
>> other operators are more generic (in: is:, the quantified operators, etc)
>>
>> I'm building a search interface for some data and documents with the
>> interface nothing more than a single search bar - basically, starting with
>> something like Calaca. My users aren't going to write ES queries directly,
>> but the power users could use qualifiers. Obviously, this isn't exactly
>> rocket science, but if there's a good example of a simple web frontend that
>> parses a simple query DSL and converts it into an ES search, I'd love to
>> use that rather than reinventing the wheel. Any language on the backend is
>> really fine, I'm not attached to anything yet so if there was Ruby or
>> Python or really whatever, it's fine by me.
>>
>> Thanks for any pointers you might have!
>>
>> -Erik
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CAKJO4n7VenJC7dj1wU3bx4BJw8UL4GRiBwzht1Uma0V3-Vwkng%40mail.gmail.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQBCXHPtLuyWRbCw37_N7u4nBR_i6EKEd3cLpM%2B-0pW8gg%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFtwNK4MmZBNGDxz5iuW8nGczP%2B927QcATam4YBLgZJNg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: github issues search query DSL

2014-08-12 Thread Nikolas Everett

I'd implement the query parser in your application and then build a the
queries and send then to Elasticsearch.  The advantage of that is that you
don't have to bounce all the Elasticsearch nodes when you upgrade your
query language.  Its what we did.  Our code isn't elegant or pretty or
anything - just regexes pulling expect syntax out of the query string and
then slapping it into filters and stuff.  Its not super hard if you do it
that way.  I'd link you to the code but its so ugly I don't advocate using
it as a starting point.

One thing, though: this is something you really really really should have
integration tests for.  Like all the way from your application's external
API to Elasticsearch.  Without it its pretty easy to break stuff.

Nik


On Tue, Aug 12, 2014 at 1:39 PM, Ivan Brusic  wrote:

> Look into the Lucene query parser, which its the syntax that the query
> string query uses. After that, look into the various Lucene contrib modules
> that extend the query syntax (span near is one).
>
> I do not think that anyone has implemented a new query parser as an
> elasticsearch plugin yet, but I could be wrong.
>
> Cheers,
>
> Ivan
> On Aug 12, 2014 1:33 PM, "Erik Paulson"  wrote:
>
>> Hello -
>>
>> Github has a pretty slick search interface for issues, complete with a
>> set of qualifiers that users can stick onto their free-form text queries.
>> They're using ES for their code search, and I'm guessing for their issues
>> as well:
>>
>> https://help.github.com/articles/searching-issues
>>
>> Is there a good library or example code somewhere that implements a
>> similar query language? Obviously some qualifiers are github-specific, but
>> other operators are more generic (in: is:, the quantified operators, etc)
>>
>> I'm building a search interface for some data and documents with the
>> interface nothing more than a single search bar - basically, starting with
>> something like Calaca. My users aren't going to write ES queries directly,
>> but the power users could use qualifiers. Obviously, this isn't exactly
>> rocket science, but if there's a good example of a simple web frontend that
>> parses a simple query DSL and converts it into an ES search, I'd love to
>> use that rather than reinventing the wheel. Any language on the backend is
>> really fine, I'm not attached to anything yet so if there was Ruby or
>> Python or really whatever, it's fine by me.
>>
>> Thanks for any pointers you might have!
>>
>> -Erik
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CAKJO4n7VenJC7dj1wU3bx4BJw8UL4GRiBwzht1Uma0V3-Vwkng%40mail.gmail.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQBCXHPtLuyWRbCw37_N7u4nBR_i6EKEd3cLpM%2B-0pW8gg%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0MezGCW%3DS%3Dd-CpgC0UZJDmHxNiOV%2BJSuPFPzn14Ov3rA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: github issues search query DSL

2014-08-12 Thread Ivan Brusic

Look into the Lucene query parser, which its the syntax that the query
string query uses. After that, look into the various Lucene contrib modules
that extend the query syntax (span near is one).

I do not think that anyone has implemented a new query parser as an
elasticsearch plugin yet, but I could be wrong.

Cheers,

Ivan
On Aug 12, 2014 1:33 PM, "Erik Paulson"  wrote:

> Hello -
>
> Github has a pretty slick search interface for issues, complete with a set
> of qualifiers that users can stick onto their free-form text queries.
> They're using ES for their code search, and I'm guessing for their issues
> as well:
>
> https://help.github.com/articles/searching-issues
>
> Is there a good library or example code somewhere that implements a
> similar query language? Obviously some qualifiers are github-specific, but
> other operators are more generic (in: is:, the quantified operators, etc)
>
> I'm building a search interface for some data and documents with the
> interface nothing more than a single search bar - basically, starting with
> something like Calaca. My users aren't going to write ES queries directly,
> but the power users could use qualifiers. Obviously, this isn't exactly
> rocket science, but if there's a good example of a simple web frontend that
> parses a simple query DSL and converts it into an ES search, I'd love to
> use that rather than reinventing the wheel. Any language on the backend is
> really fine, I'm not attached to anything yet so if there was Ruby or
> Python or really whatever, it's fine by me.
>
> Thanks for any pointers you might have!
>
> -Erik
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAKJO4n7VenJC7dj1wU3bx4BJw8UL4GRiBwzht1Uma0V3-Vwkng%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQBCXHPtLuyWRbCw37_N7u4nBR_i6EKEd3cLpM%2B-0pW8gg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

github issues search query DSL

2014-08-12 Thread Erik Paulson

Hello -

Github has a pretty slick search interface for issues, complete with a set
of qualifiers that users can stick onto their free-form text queries.
They're using ES for their code search, and I'm guessing for their issues
as well:

https://help.github.com/articles/searching-issues

Is there a good library or example code somewhere that implements a similar
query language? Obviously some qualifiers are github-specific, but other
operators are more generic (in: is:, the quantified operators, etc)

I'm building a search interface for some data and documents with the
interface nothing more than a single search bar - basically, starting with
something like Calaca. My users aren't going to write ES queries directly,
but the power users could use qualifiers. Obviously, this isn't exactly
rocket science, but if there's a good example of a simple web frontend that
parses a simple query DSL and converts it into an ES search, I'd love to
use that rather than reinventing the wheel. Any language on the backend is
really fine, I'm not attached to anything yet so if there was Ruby or
Python or really whatever, it's fine by me.

Thanks for any pointers you might have!

-Erik

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKJO4n7VenJC7dj1wU3bx4BJw8UL4GRiBwzht1Uma0V3-Vwkng%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: About elasticsearch.org

I just hear second hand that the outage is pretty large.

On Tue, Aug 12, 2014 at 10:14 AM, Jack Park  wrote:
> It appears that liquidweb (their host) has some problems.  I and a
> friend in Canada can open it on my cell phone, but nobody I know
> around here can raise it on some other networks.
>
> On Tue, Aug 12, 2014 at 9:58 AM, Antonio Augusto Santos
>  wrote:
>> Working fine for me (in Brazil).
>>
>>
>> On Tuesday, August 12, 2014 1:57:48 PM UTC-3, Jack Park wrote:
>>>
>>> Just curious:
>>> on all browsers here in silicon valley, I cannot raise any
>>> elasticsearch.org
>>>
>>> Is it just me (or comcast?)
>>> Other websites appear fine.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/ee9afb15-2e69-400c-a21c-2017d3d818ea%40googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAH6s0fwTcaAuAjBrB_8GnoBwxijBLvhbGOwuKV%2B37_6Tq958rw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: About elasticsearch.org

It appears that liquidweb (their host) has some problems.  I and a
friend in Canada can open it on my cell phone, but nobody I know
around here can raise it on some other networks.

On Tue, Aug 12, 2014 at 9:58 AM, Antonio Augusto Santos
 wrote:
> Working fine for me (in Brazil).
>
>
> On Tuesday, August 12, 2014 1:57:48 PM UTC-3, Jack Park wrote:
>>
>> Just curious:
>> on all browsers here in silicon valley, I cannot raise any
>> elasticsearch.org
>>
>> Is it just me (or comcast?)
>> Other websites appear fine.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/ee9afb15-2e69-400c-a21c-2017d3d818ea%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAH6s0fyJRuVKoJpDZm%3DC7k8ym9QXYNP%3DpJE%2Bvu1aTGFk%2B2ww%2BA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: About elasticsearch.org

2014-08-12 Thread Antonio Augusto Santos

Working fine for me (in Brazil).

On Tuesday, August 12, 2014 1:57:48 PM UTC-3, Jack Park wrote:
>
> Just curious: 
> on all browsers here in silicon valley, I cannot raise any 
> elasticsearch.org 
>
> Is it just me (or comcast?) 
> Other websites appear fine. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/ee9afb15-2e69-400c-a21c-2017d3d818ea%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

About elasticsearch.org

Just curious:
on all browsers here in silicon valley, I cannot raise any elasticsearch.org

Is it just me (or comcast?)
Other websites appear fine.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAH6s0fxje2oHjikafsQcA%3DWKE9WeoQ0R-Hocc5SucCd2CRR%3DhQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

New version of Kibana in the works?

2014-08-12 Thread Antonio Augusto Santos

This one is for the devs, and Rashid in special: there is any new version 
of Kibana in the works?
I'm asking this because I'm about to start a project in my company for log 
management, and there are some requisites to it (user separation, event 
correlation, histogram to compare two values, and so on).

So, any changes of these functionalities landing on Kibana 4.0? ;)

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/2780b7a3-6312-4883-8a02-42b5aeefd88d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Failed to perform snapshot (index files)]; nested: FileNotFoundException

2014-08-12 Thread Aleh Aleshka

I upgraded to 1.3.1 and one of the indexes seems to backup ok now, though 
the other one is still giving the error.

On Monday, August 11, 2014 8:40:44 PM UTC+3, Aleh Aleshka wrote:
>
> Hello
> I have a 1.2.2 cluster of 6 nodes with several indexes configured with 2 
> to 4 replicas.
> I'm trying to perform a snapshot but it completes only partially with 
> failures like this:
>
> CreateSnapshotResponse[snapshotInfo=SnapshotInfo[name=2014-08-11-16-31-04,
> state=PARTIAL,reason=,indices=Object[][{my_idx3,my_idx2}],
> startTime=1407774870154,endTime=1407775114709,totalShards=17,
> successfulShards=14,
> shardFailures=Object[][{
> [my_idx2][4] failed, reason [IndexShardSnapshotFailedException[[my_idx2][4
> ] Failed to perform snapshot (index files)]; nested: FileNotFoundException
> [/home/shared_dir/indices/my_idx2/4/__0 (No such file or directory)]; ],
> [my_idx2][3] failed, reason [IndexShardSnapshotFailedException[[my_idx2][3
> ] Failed to perform snapshot (index files)]; nested: FileNotFoundException
> [/home/shared_dir/indices/my_idx2/3/__0 (No such file or directory)]; ],
> [my_idx3][0] failed, reason [IndexShardSnapshotFailedException[[my_idx3][0
> ] Failed to perform snapshot (index files)]; nested: FileNotFoundException
> [/home/shared_dir/indices/my_idx3/0/__0 (No such file or directory)]; 
> ]}]],
> headers=,remoteAddress=inet[masternode/redacted:9300]]
>
> I used  CheckIndex to verify that those shards on all nodes do not have 
> any problems.
> What might be the reason for that?
>
>
> Aleh
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1385c597-94b1-4372-9c7b-a89fbd0cb4dd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Embedded ElasticSearch On Java

The default cluster name is "elasticsearch". Changing it in your code
works


On Tue, Aug 12, 2014 at 9:33 PM, Vivek Sachdeva <
vivek.sachd...@intelligrape.com> wrote:

> Your code works if you dont add cluster name to it. Tried with Java
> this time.. :)
>
>
> On Tue, Aug 12, 2014 at 7:47 PM, Kfeenz  wrote:
>
>> @Jorg,
>>
>> Thanks for the advice, I will make sure that I do so during actual
>> implementation, but this is purely for testing the connection.. Also, I see
>> a client.close() and a client.threadPool().shutdown(), but I do not see a
>> client.threadPool().close(). I am using ES v1.3.1.
>>
>> @ Vivek,
>>
>> I am not sure how you were able to use 'localhost' vise "localhost". Java
>> complains about an invalid character constant because 'localhost' is not a
>> character but a String...
>>
>> My current code is as follows... with still no luck...
>>
>> Settings settings = ImmutableSettings.settingsBuilder().put("cluster.name",
>> "mycluster").build();
>>
>> Client client = new TransportClient(settings).addTransportAddress(new
>> InetSocketTransportAddress("localhost", 9300));
>>
>> ClusterStatsRequestBuilder builder =
>> client.admin().cluster().prepareClusterStats();
>>
>> ClusterStatsResponse response = builder.execute().actionGet(); // fails
>> on execute... NoNodeAvailableException
>>
>> assertEquals("mycluster", response.getClusterName()); // never gets to
>> this point
>>
>> NoNodeAvailableException: None of the configured nodes are available []
>>
>> If I add a setting to the settings object
>>
>> .put("client.transport.sniff", true);
>>
>> I get a different error - [org.elasticsearch.client.transport] [Argus]
>> failed to get local cluster state info for [#transport#-1]...
>>
>> I can query the cluster using 
>> *http://localhost:9200/_cluster/health?pretty=true
>> * which returns
>>
>> {
>>   "cluster_name" : "mycluster",
>>   "status" : "green",
>>   "timed_out" : false,
>>   "number_of_nodes" : 1,
>>   "number_of_data_nodes" : 1,
>>   "active_primary_shards" : 0,
>>   "active_shards" : 0,
>>   "relocating_shards" : 0,
>>   "initializing_shards" : 0,
>>   "unassigned_shards" : 0
>> }
>>
>> I am on Windows 7 64-bit.
>> I am using Java 1.7_u55.
>> I am using ES version 1.3.1.
>> I have included in my pom.xml:
>>   - elasticsearch-1.3.1.jar
>>   - lucene-core-4.9.0.jar
>>
>> Any other suggestions are greatly appreciated.
>>
>>
>>
>> On Tuesday, August 12, 2014 5:45:16 AM UTC-4, Vivek Sachdeva wrote:
>>>
>>> Replace
>>>
>>> .setTransportAddress(new InetSocketTransportAddress("localhost",
>>> 9300));
>>>
>>> with
>>>
>>> .addTransportAddress(new InetSocketTransportAddress('localhost', 9300)).
>>>
>>> And I guess if you dont give cluster name, it automatically joins the
>>> default cluster.
>>>
>>> I tried the code that you provided and changed above mentioned code. It
>>> works on my end. Can you try it?
>>>
>>> On Monday, August 11, 2014 11:34:43 PM UTC+5:30, Kfeenz wrote:

 So I am very new to elasticsearch... so I apologize in advance..

 I started a local instance of elasticsearch and I am trying to connect
 to it through the Java API.

 I was under the impression that the transport client was for remote
 clients?

 I tried:

 @Test
 public void testIndexResponse() {

   Client client = new TransportClient().setTransportAddress(new
 InetSocketTransportAddress("localhost", 9300));

   String json = "{" +
 "\"user\":\"kimchy\"," +


 "\"postDate\":\"2013-01-30\"," +
 "\"message\":\"trying out Elasticsearch\"" +


 "}";

   IndexResponse response = client.prepareIndex("twitter", "tweet")


 .setSource(json)
 .execute()
 .actionGet();


   client.close();

   System.out.println(response.getIndex());
 }

 I receive org.elasticsearch.client.transport.NoNodeAvailableException:
 None of the configured nodes are available: [].



 On Monday, August 11, 2014 1:19:06 PM UTC-4, Vivek Sachdeva wrote:
>
> Have you tried using transport client for connecting...
>
> On Monday, August 11, 2014 10:26:29 PM UTC+5:30, Kfeenz wrote:
>>
>> All,
>>
>> I know this post is old, but I continue to have an issue with this...
>>
>> I get an NoSuchMethodError: org.apache.log4j.Logger.isTraceEnabled()Z
>> exception when I run
>>
>> Node node = NodeBuilder.nodeBuilder().local(true).node(); //
>> exception thrown here...
>> Client client = node.client();
>>
>> I have tried including several different slf4j and log4j libraries...
>>
>> Right now I have defined in my pom.xml:
>>
>> slf4j-api-1.7.7.jar
>> slf4j-ext-1.7.7.jar
>> log4j-core-2.0.jar
>> log4j-api-2.0.jar
>>
>> Any suggestions?
>> log4j-slf4j-impl-2.0.jar
>>
>> On Wednesday, Nov

Re: Embedded ElasticSearch On Java

Your code works if you dont add cluster name to it. Tried with Java
this time.. :)


On Tue, Aug 12, 2014 at 7:47 PM, Kfeenz  wrote:

> @Jorg,
>
> Thanks for the advice, I will make sure that I do so during actual
> implementation, but this is purely for testing the connection.. Also, I see
> a client.close() and a client.threadPool().shutdown(), but I do not see a
> client.threadPool().close(). I am using ES v1.3.1.
>
> @ Vivek,
>
> I am not sure how you were able to use 'localhost' vise "localhost". Java
> complains about an invalid character constant because 'localhost' is not a
> character but a String...
>
> My current code is as follows... with still no luck...
>
> Settings settings = ImmutableSettings.settingsBuilder().put("cluster.name",
> "mycluster").build();
>
> Client client = new TransportClient(settings).addTransportAddress(new
> InetSocketTransportAddress("localhost", 9300));
>
> ClusterStatsRequestBuilder builder =
> client.admin().cluster().prepareClusterStats();
>
> ClusterStatsResponse response = builder.execute().actionGet(); // fails on
> execute... NoNodeAvailableException
>
> assertEquals("mycluster", response.getClusterName()); // never gets to
> this point
>
> NoNodeAvailableException: None of the configured nodes are available []
>
> If I add a setting to the settings object
>
> .put("client.transport.sniff", true);
>
> I get a different error - [org.elasticsearch.client.transport] [Argus]
> failed to get local cluster state info for [#transport#-1]...
>
> I can query the cluster using 
> *http://localhost:9200/_cluster/health?pretty=true
> * which returns
>
> {
>   "cluster_name" : "mycluster",
>   "status" : "green",
>   "timed_out" : false,
>   "number_of_nodes" : 1,
>   "number_of_data_nodes" : 1,
>   "active_primary_shards" : 0,
>   "active_shards" : 0,
>   "relocating_shards" : 0,
>   "initializing_shards" : 0,
>   "unassigned_shards" : 0
> }
>
> I am on Windows 7 64-bit.
> I am using Java 1.7_u55.
> I am using ES version 1.3.1.
> I have included in my pom.xml:
>   - elasticsearch-1.3.1.jar
>   - lucene-core-4.9.0.jar
>
> Any other suggestions are greatly appreciated.
>
>
>
> On Tuesday, August 12, 2014 5:45:16 AM UTC-4, Vivek Sachdeva wrote:
>>
>> Replace
>>
>> .setTransportAddress(new InetSocketTransportAddress("localhost", 9300));
>>
>> with
>>
>> .addTransportAddress(new InetSocketTransportAddress('localhost', 9300)).
>>
>> And I guess if you dont give cluster name, it automatically joins the
>> default cluster.
>>
>> I tried the code that you provided and changed above mentioned code. It
>> works on my end. Can you try it?
>>
>> On Monday, August 11, 2014 11:34:43 PM UTC+5:30, Kfeenz wrote:
>>>
>>> So I am very new to elasticsearch... so I apologize in advance..
>>>
>>> I started a local instance of elasticsearch and I am trying to connect
>>> to it through the Java API.
>>>
>>> I was under the impression that the transport client was for remote
>>> clients?
>>>
>>> I tried:
>>>
>>> @Test
>>> public void testIndexResponse() {
>>>
>>>   Client client = new TransportClient().setTransportAddress(new
>>> InetSocketTransportAddress("localhost", 9300));
>>>
>>>   String json = "{" +
>>> "\"user\":\"kimchy\"," +
>>>
>>> "\"postDate\":\"2013-01-30\"," +
>>> "\"message\":\"trying out Elasticsearch\"" +
>>>
>>> "}";
>>>
>>>   IndexResponse response = client.prepareIndex("twitter", "tweet")
>>>
>>> .setSource(json)
>>> .execute()
>>> .actionGet();
>>>
>>>
>>>   client.close();
>>>
>>>   System.out.println(response.getIndex());
>>> }
>>>
>>> I receive org.elasticsearch.client.transport.NoNodeAvailableException:
>>> None of the configured nodes are available: [].
>>>
>>>
>>>
>>> On Monday, August 11, 2014 1:19:06 PM UTC-4, Vivek Sachdeva wrote:

 Have you tried using transport client for connecting...

 On Monday, August 11, 2014 10:26:29 PM UTC+5:30, Kfeenz wrote:
>
> All,
>
> I know this post is old, but I continue to have an issue with this...
>
> I get an NoSuchMethodError: org.apache.log4j.Logger.isTraceEnabled()Z
> exception when I run
>
> Node node = NodeBuilder.nodeBuilder().local(true).node(); //
> exception thrown here...
> Client client = node.client();
>
> I have tried including several different slf4j and log4j libraries...
>
> Right now I have defined in my pom.xml:
>
> slf4j-api-1.7.7.jar
> slf4j-ext-1.7.7.jar
> log4j-core-2.0.jar
> log4j-api-2.0.jar
>
> Any suggestions?
> log4j-slf4j-impl-2.0.jar
>
> On Wednesday, November 30, 2011 6:55:57 PM UTC-5, Sam wrote:
>>
>> All
>>
>> I have a webapp (JAVA) with a list of items on the page. All these
>> items are generated on html with session variables. I want to use
>> ElasticSearch on the back end to do a search for items with different
>> search criteria. Is there a sample I can use it as a reso

Unallocated shards with empty nodes

2014-08-12 Thread Christopher Ambler




As you can see by the image snapped from 'head,' I have unallocated shards 
while a couple of nodes are empty. This happened when I recycled some 
nodes. The last two indexes are freshly-created and look good. (5 shards, 5 
replicas + master).

Nothing I do gets those unallocated shards back on a node.

Anyone have a clue as to what I'm doing wrong?


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/7936bcbf-a1f6-421a-9935-4618e056daf2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: A fresh variant of MapperParsingException

2014-08-12 Thread Jun Ohtani

hi,

you should not remove "core".
please try to add that.
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-put-mapping.html

2014年8月12日火曜日、Jack Parkさんは書きました:

> I created a gist for the mappings I am sending through node-es, with a
> returned error:
> MapperParsingException[Root type mapping not empty after parsing!
>
> The gist is at
> https://gist.github.com/KnowledgeGarden/b965b7e78f19f9be9025
>
> Note, that if I remove the upper portion of the json:
> "topics": {
> "mappings": {
> "core": {
> and just send in from "properties":, the same error occurs.
>
> The exact code which sends mappings into node-es is found here:
> https://github.com/KnowledgeGarden/TQTopicMap/blob/master/lib/persist/es.js
>
> I'm wondering if anyone recognizes an error in my mapping. I
> deliberately include the index and type in the mapping. It seems hard
> to find examples of schema mappings that do what I am trying to do.
>
> Thanks!
>
> Here is the error message:
> ESClient-1 Error: {"error":"MapperParsingException[mapping [topics]];
> nested: Ma
> pperParsingException[Root type mapping not empty after parsing! Remaining
> fields
> : [mappings : {core={properties={inOf={index=not_analyzed, store=true,
> type=stri
> ng}, _ver={index=not_analyzed, store=true, type=string},
> tpC={index=not_analyzed
> , store=true, type=long}, lox={index=not_analyzed, store=true,
> type=string}, srt
> Dt={index=not_analyzed, store=true, type=long}, url={index=not_analyzed,
> store=t
> rue, type=string}, lists={properties={body={index=analyzed, store=true,
> type=str
> ing}, sbOf={index=not_analyzed, store=true, type=string},
> details={index=analyze
> d, store=true, type=string}, psi={index=not_analyzed, store=true,
> type=string},
> label={index=analyzed, store=true, type=string}, subj={index=analyzed,
> store=tru
> e, type=string}}]]; ","status":400}
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com .
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAH6s0fxkx%3D0ZpDM-qAEi4RREKKVubUcX%3DVSPSz5mwGTxYuWNNw%40mail.gmail.com
> .
> For more options, visit https://groups.google.com/d/optout.
>


-- 
---
Jun Ohtani
blog : http://blog.johtani.info

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAPW8A5zBLQNyrwX0HZwc6CECYV6H%3DqHUS5aKuy%2BCouO6b%2B9EtQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Uneven Primary Shard Distribution in Cluster

Latency is an issue, you are right.  But that is not related to
primary/replica distribution.

If cluster state changes, e.g. new field names arrive, the master must be
reached very quickly for update, and the master pushes out the new state to
all nodes. It is crucial that state propagation must complete
instantaneously before indexing continues.

At the moment, there is no good solution for cross-continental networking,
since ES requires low latency networking. The best I can imagine is to set
up two clusters and sync them with an extra tool over a high latency line.

Jörg


On Tue, Aug 12, 2014 at 5:15 PM, Andrew Ruslander <
andrew.ruslan...@gmail.com> wrote:

> Thanks for the response, Jörg.  My only concern is that what if the two
> sides are located some non-trivial geographic distance from each other?  If
> all the primaries live on SideB and there is a large quantity of updates
> coming into a node on SideA, doesn't it have to forward all that to the
> node containing the primary on SideB?  I worry about latency there.
>
>  - Andrew
>
>
> On Tuesday, August 12, 2014 10:49:55 AM UTC-4, Jörg Prante wrote:
>
>> That's ok, this is how Elasticsearch works. There is no need to randomize
>> or shuffle primaries. They have exactly the same work to do as replicas.
>> Replicas are promoted to primaries automatically on demand.
>>
>> Jörg
>>
>>
>> On Tue, Aug 12, 2014 at 4:24 PM, Andrew Ruslander 
>> wrote:
>>
>>> I have a five node Elasticsearch cluster set up so that two nodes are in
>>> one zone and the other three nodes are in a different zone (let's call them
>>> SideA and SideB) via use of the forced awareness attributes.  I also have a
>>> sixth node that has Logstash on it.  Logstash is outputting to one of the
>>> two nodes in SideA.  However, when I use Marvel to view the shard
>>> allocation across my five node cluster, I see that probably 95% of the
>>> primary shards are all on SideB.  SideA is almost exclusively replicas.
>>>  Just out of curiosity, I changed Logstash to output to a node in SideB,
>>> but the primaries continued to be allocated to SideB only.
>>>
>>> So my question is...is this expected behavior?  What would cause the
>>> primary shards to be allocated only to one side?  Is it because it has
>>> three nodes versus the two nodes on SideA?  Is something else afoot here?
>>>
>>> - Andrew
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>>
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/elasticsearch/dba612d0-adb9-4bca-8c67-3b35e9236bc7%
>>> 40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/76d5de6f-4526-41cf-970e-0b54253bfea6%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFeNTzsdtHiT%2Bt0R%2BH2U056gKQEeYRH7-PHyiXau2qCYw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Uneven Primary Shard Distribution in Cluster

2014-08-12 Thread Andrew Ruslander

Thanks for the response, Jörg.  My only concern is that what if the two 
sides are located some non-trivial geographic distance from each other?  If 
all the primaries live on SideB and there is a large quantity of updates 
coming into a node on SideA, doesn't it have to forward all that to the 
node containing the primary on SideB?  I worry about latency there.

 - Andrew

On Tuesday, August 12, 2014 10:49:55 AM UTC-4, Jörg Prante wrote:
>
> That's ok, this is how Elasticsearch works. There is no need to randomize 
> or shuffle primaries. They have exactly the same work to do as replicas. 
> Replicas are promoted to primaries automatically on demand.
>
> Jörg
>
>
> On Tue, Aug 12, 2014 at 4:24 PM, Andrew Ruslander  > wrote:
>
>> I have a five node Elasticsearch cluster set up so that two nodes are in 
>> one zone and the other three nodes are in a different zone (let's call them 
>> SideA and SideB) via use of the forced awareness attributes.  I also have a 
>> sixth node that has Logstash on it.  Logstash is outputting to one of the 
>> two nodes in SideA.  However, when I use Marvel to view the shard 
>> allocation across my five node cluster, I see that probably 95% of the 
>> primary shards are all on SideB.  SideA is almost exclusively replicas. 
>>  Just out of curiosity, I changed Logstash to output to a node in SideB, 
>> but the primaries continued to be allocated to SideB only.
>>
>> So my question is...is this expected behavior?  What would cause the 
>> primary shards to be allocated only to one side?  Is it because it has 
>> three nodes versus the two nodes on SideA?  Is something else afoot here?
>>
>> - Andrew
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/dba612d0-adb9-4bca-8c67-3b35e9236bc7%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/76d5de6f-4526-41cf-970e-0b54253bfea6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Plugin point for adding custom based query

2014-08-12 Thread vineeth mohan

Hi ,


Is there any plugin point like river or analyzer to add my own custom made
Lucene query ?

Thanks
   Vineeth

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3D%2BcZxnYezqzwUWHGs8R%2BC36hg5mN4hMnHhstvguS63Tw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch inserting date type documents as UTC timezone datetime while indexing

2014-08-12 Thread Subhadip Bagui

Hi,

Can someone please give me a hint, I'm having trouble getting a solution 
for this.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d42cb94b-681d-4bd3-bf21-d955cd0af729%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Embedded ElasticSearch On Java

Yes, it's client.threadPool().shutdown().

Jörg


On Tue, Aug 12, 2014 at 4:17 PM, Kfeenz  wrote:

> @Jorg,
>
> Thanks for the advice, I will make sure that I do so during actual
> implementation, but this is purely for testing the connection.. Also, I see
> a client.close() and a client.threadPool().shutdown(), but I do not see a
> client.threadPool().close(). I am using ES v1.3.1.
>
> @ Vivek,
>
> I am not sure how you were able to use 'localhost' vise "localhost". Java
> complains about an invalid character constant because 'localhost' is not a
> character but a String...
>
> My current code is as follows... with still no luck...
>
> Settings settings = ImmutableSettings.settingsBuilder().put("cluster.name",
> "mycluster").build();
>
> Client client = new TransportClient(settings).addTransportAddress(new
> InetSocketTransportAddress("localhost", 9300));
>
> ClusterStatsRequestBuilder builder =
> client.admin().cluster().prepareClusterStats();
>
> ClusterStatsResponse response = builder.execute().actionGet(); // fails on
> execute... NoNodeAvailableException
>
> assertEquals("mycluster", response.getClusterName()); // never gets to
> this point
>
> NoNodeAvailableException: None of the configured nodes are available []
>
> If I add a setting to the settings object
>
> .put("client.transport.sniff", true);
>
> I get a different error - [org.elasticsearch.client.transport] [Argus]
> failed to get local cluster state info for [#transport#-1]...
>
> I can query the cluster using 
> *http://localhost:9200/_cluster/health?pretty=true
> * which returns
>
> {
>   "cluster_name" : "mycluster",
>   "status" : "green",
>   "timed_out" : false,
>   "number_of_nodes" : 1,
>   "number_of_data_nodes" : 1,
>   "active_primary_shards" : 0,
>   "active_shards" : 0,
>   "relocating_shards" : 0,
>   "initializing_shards" : 0,
>   "unassigned_shards" : 0
> }
>
> I am on Windows 7 64-bit.
> I am using Java 1.7_u55.
> I am using ES version 1.3.1.
> I have included in my pom.xml:
>   - elasticsearch-1.3.1.jar
>   - lucene-core-4.9.0.jar
>
> Any other suggestions are greatly appreciated.
>
>
>
> On Tuesday, August 12, 2014 5:45:16 AM UTC-4, Vivek Sachdeva wrote:
>>
>> Replace
>>
>> .setTransportAddress(new InetSocketTransportAddress("localhost", 9300));
>>
>> with
>>
>> .addTransportAddress(new InetSocketTransportAddress('localhost', 9300)).
>>
>> And I guess if you dont give cluster name, it automatically joins the
>> default cluster.
>>
>> I tried the code that you provided and changed above mentioned code. It
>> works on my end. Can you try it?
>>
>> On Monday, August 11, 2014 11:34:43 PM UTC+5:30, Kfeenz wrote:
>>>
>>> So I am very new to elasticsearch... so I apologize in advance..
>>>
>>> I started a local instance of elasticsearch and I am trying to connect
>>> to it through the Java API.
>>>
>>> I was under the impression that the transport client was for remote
>>> clients?
>>>
>>> I tried:
>>>
>>> @Test
>>> public void testIndexResponse() {
>>>
>>>   Client client = new TransportClient().setTransportAddress(new
>>> InetSocketTransportAddress("localhost", 9300));
>>>
>>>   String json = "{" +
>>> "\"user\":\"kimchy\"," +
>>> "\"postDate\":\"2013-01-30\"," +
>>> "\"message\":\"trying out Elasticsearch\"" +
>>> "}";
>>>
>>>   IndexResponse response = client.prepareIndex("twitter", "tweet")
>>> .setSource(json)
>>> .execute()
>>> .actionGet();
>>>
>>>
>>>   client.close();
>>>
>>>   System.out.println(response.getIndex());
>>> }
>>>
>>> I receive org.elasticsearch.client.transport.NoNodeAvailableException:
>>> None of the configured nodes are available: [].
>>>
>>>
>>>
>>> On Monday, August 11, 2014 1:19:06 PM UTC-4, Vivek Sachdeva wrote:

 Have you tried using transport client for connecting...

 On Monday, August 11, 2014 10:26:29 PM UTC+5:30, Kfeenz wrote:
>
> All,
>
> I know this post is old, but I continue to have an issue with this...
>
> I get an NoSuchMethodError: org.apache.log4j.Logger.isTraceEnabled()Z
> exception when I run
>
> Node node = NodeBuilder.nodeBuilder().local(true).node(); //
> exception thrown here...
> Client client = node.client();
>
> I have tried including several different slf4j and log4j libraries...
>
> Right now I have defined in my pom.xml:
>
> slf4j-api-1.7.7.jar
> slf4j-ext-1.7.7.jar
> log4j-core-2.0.jar
> log4j-api-2.0.jar
>
> Any suggestions?
> log4j-slf4j-impl-2.0.jar
>
> On Wednesday, November 30, 2011 6:55:57 PM UTC-5, Sam wrote:
>>
>> All
>>
>> I have a webapp (JAVA) with a list of items on the page. All these
>> items are generated on html with session variables. I want to use
>> ElasticSearch on the back end to do a search for items with different
>> search criteria. Is there a sample I can use it as a resource?
>>
>> Thanks
>>
>>  --
> You

A fresh variant of MapperParsingException

I created a gist for the mappings I am sending through node-es, with a
returned error:
MapperParsingException[Root type mapping not empty after parsing!

The gist is at
https://gist.github.com/KnowledgeGarden/b965b7e78f19f9be9025

Note, that if I remove the upper portion of the json:
"topics": {
"mappings": {
"core": {
and just send in from "properties":, the same error occurs.

The exact code which sends mappings into node-es is found here:
https://github.com/KnowledgeGarden/TQTopicMap/blob/master/lib/persist/es.js

I'm wondering if anyone recognizes an error in my mapping. I
deliberately include the index and type in the mapping. It seems hard
to find examples of schema mappings that do what I am trying to do.

Thanks!

Here is the error message:
ESClient-1 Error: {"error":"MapperParsingException[mapping [topics]]; nested: Ma
pperParsingException[Root type mapping not empty after parsing! Remaining fields
: [mappings : {core={properties={inOf={index=not_analyzed, store=true, type=stri
ng}, _ver={index=not_analyzed, store=true, type=string}, tpC={index=not_analyzed
, store=true, type=long}, lox={index=not_analyzed, store=true, type=string}, srt
Dt={index=not_analyzed, store=true, type=long}, url={index=not_analyzed, store=t
rue, type=string}, lists={properties={body={index=analyzed, store=true, type=str
ing}, sbOf={index=not_analyzed, store=true, type=string}, details={index=analyze
d, store=true, type=string}, psi={index=not_analyzed, store=true, type=string},
label={index=analyzed, store=true, type=string}, subj={index=analyzed, store=tru
e, type=string}}]]; ","status":400}

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAH6s0fxkx%3D0ZpDM-qAEi4RREKKVubUcX%3DVSPSz5mwGTxYuWNNw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Uneven Primary Shard Distribution in Cluster

That's ok, this is how Elasticsearch works. There is no need to randomize
or shuffle primaries. They have exactly the same work to do as replicas.
Replicas are promoted to primaries automatically on demand.

Jörg


On Tue, Aug 12, 2014 at 4:24 PM, Andrew Ruslander <
andrew.ruslan...@gmail.com> wrote:

> I have a five node Elasticsearch cluster set up so that two nodes are in
> one zone and the other three nodes are in a different zone (let's call them
> SideA and SideB) via use of the forced awareness attributes.  I also have a
> sixth node that has Logstash on it.  Logstash is outputting to one of the
> two nodes in SideA.  However, when I use Marvel to view the shard
> allocation across my five node cluster, I see that probably 95% of the
> primary shards are all on SideB.  SideA is almost exclusively replicas.
>  Just out of curiosity, I changed Logstash to output to a node in SideB,
> but the primaries continued to be allocated to SideB only.
>
> So my question is...is this expected behavior?  What would cause the
> primary shards to be allocated only to one side?  Is it because it has
> three nodes versus the two nodes on SideA?  Is something else afoot here?
>
> - Andrew
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/dba612d0-adb9-4bca-8c67-3b35e9236bc7%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoESRqSn5u1TEkWfoqNgnPmOgfy%2BbO5%2Bhv3gJ%2BDSZoBaCg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Can't delete Index!

There is one node with zen unicast that can not connect. I do not know how
to find more about that, it seems an EC2 issue.

Except you are being attacked in vain by the scripting vulnerability, there
are only syntax errors in configuration and queries... you should be able
to fix that.

Jörg



On Tue, Aug 12, 2014 at 4:25 PM, Sam2014  wrote:

> Just wondering, will remove/re-install get rid of the stagnant index since
> I plan on upgrading from 1.2.1 to 1.2.2?
>
>
> On Tuesday, August 12, 2014 9:42:10 AM UTC-4, Sam2014 wrote:
>>
>> The link doesn't work.
>> I am not sure about the difference, I have always deleted the indices the
>> same way, curl -XPUT http... //to create, and curl -XDelete /http:...
>> I did not create any aliases, I am just using Get _aliases to see whats
>> still on the node.
>> Did the log tell you anything, there are some weird errors in there that
>> I don't know how to interpret, a quick look there may tell you something.
>>
>> On Tuesday, August 12, 2014 3:04:39 AM UTC-4, Jörg Prante wrote:
>>>
>>> So you are wrestling with aliases. You can not delete aliases by file
>>> system operations. Have you checked
>>>
>>> http://www.elasticsearch.org/guide/en/elasticsearch/
>>> reference/current/indices-aliases.html#deleting
>>>
>>> for deleting aliases?
>>>
>>> Jörg
>>>
>>>
>>> On Tue, Aug 12, 2014 at 4:10 AM, Sam2014  wrote:
>>>
 ///Attached is the cluster log that covers the whole sequence from
 shutting the cluster and re-activating it.

 //Show me current state of cluster
 {
"cluster_name": "rexCluster",
"status": "red",
"timed_out": false,
"number_of_nodes": 2,
"number_of_data_nodes": 2,
"active_primary_shards": 7,
"active_shards": 7,
"relocating_shards": 0,
"initializing_shards": 2,
"unassigned_shards": 21
 }

 //Show me indices on the cluster

 GET _aliases
 {
"foo": {
   "aliases": {}
},
"mediregs": {
   "aliases": {}
},
"twitter": {
   "aliases": {}
}
 }



 //Shut down the cluster

 POST /_cluster/nodes/_shutdown
 {
"cluster_name": "rexCluster",
"nodes": {
   "_vPEfmc9TruIpugk4NWleg": {
  "name": "node 1"
   },
   "DB94rWiiSvS9HUTXwI7GTw": {
  "name": "node 2"
   }
}
 }

  //Go into the fs and delete files under /data

 ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0$
 ls

 *indices*  node.lock  *_state*

 ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0$
 cd indices/

 ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$
 ls

 *foo*  *mediregs*  *twitter*

 ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$
 sudo rm -rf foo

 ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$
 sudo rm -rf mediregs

 ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$
 sudo rm -rf twitter

 ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$
 ls

 ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/
 rexCluster/nodes/0/indices$
 //Restart nodes 1 &2 - should be in log file

 Get _aliases //Again - they came back!!
 {
"foo": {
   "aliases": {}
},
"mediregs": {
   "aliases": {}
},
"twitter": {
   "aliases": {}
}
 }


 //Ok - Try to delete index
 DELETE /mediregs

 //takes little while and comes back
 {
"acknowledged": false
 }




 On Monday, August 11, 2014 7:12:22 PM UTC-4, Jörg Prante wrote:

> Can you post your detailed steps and debug logs?
>
> Make also sure there is no third node - this looks like ES is shipping
> shards to a third node from which the two nodes are recovering from.
>
> Jörg
>
>
> On Tue, Aug 12, 2014 at 1:06 AM, Sam2014  wrote:
>
>> I stopped node 1, then I stopped node 2. I disabled the cluster,
>> deleted files from nodes, Made sure no Indices on either node. I 
>> re-enabled
>> cluster in yml, restarted cluster, and the index came back!
>>
>>
>> On Monday, August 11, 2014 6:16:42 PM UTC-4, Jörg Prante wrote:
>>
>>> There is no vicious circle. First you must stop the cluster. Then,
>>> after processes have exited, continue with file operations. After that, 
>>> you
>>> can start the cluster nodes again.
>>>
>>> Jörg
>>>
>>>
>>> On Mon, Aug 11, 2014 at 11:49 PM, Sam2014 
>>> wrote:
>>>
 Oops, When I restarted the cluster/nodes,

Re: Embedded ElasticSearch On Java

2014-08-12 Thread Kfeenz

Yes I receive back a TransportClient back from the call client = new 
TransportClient()

In debug I see that the nodeService.clusterName.value = "mycluster" as 
expected.

But it still fails on the execute() call

On Tuesday, August 12, 2014 10:29:30 AM UTC-4, Vivek Sachdeva wrote:
>
> Actually I am using groovy... So 'localhost' and "localhost" are same fr 
> me... Are you getting object of transport client in your code...
>
>
> On Tuesday, August 12, 2014, Kfeenz > 
> wrote:
>
>> @Jorg,
>>
>> Thanks for the advice, I will make sure that I do so during actual 
>> implementation, but this is purely for testing the connection.. Also, I see 
>> a client.close() and a client.threadPool().shutdown(), but I do not see a 
>> client.threadPool().close(). I am using ES v1.3.1.
>>
>> @ Vivek,
>>
>> I am not sure how you were able to use 'localhost' vise "localhost". Java 
>> complains about an invalid character constant because 'localhost' is not a 
>> character but a String...
>>
>> My current code is as follows... with still no luck...
>>
>> Settings settings = ImmutableSettings.settingsBuilder().put("cluster.name", 
>> "mycluster").build();
>>
>> Client client = new TransportClient(settings).addTransportAddress(new 
>> InetSocketTransportAddress("localhost", 9300));
>>
>> ClusterStatsRequestBuilder builder = 
>> client.admin().cluster().prepareClusterStats();
>>
>> ClusterStatsResponse response = builder.execute().actionGet(); // fails 
>> on execute... NoNodeAvailableException
>>
>> assertEquals("mycluster", response.getClusterName()); // never gets to 
>> this point
>>
>> NoNodeAvailableException: None of the configured nodes are available []
>>
>> If I add a setting to the settings object
>>
>> .put("client.transport.sniff", true);
>>
>> I get a different error - [org.elasticsearch.client.transport] [Argus] 
>> failed to get local cluster state info for [#transport#-1]...
>>
>> I can query the cluster using 
>> *http://localhost:9200/_cluster/health?pretty=true 
>> * which returns
>>
>> {
>>   "cluster_name" : "mycluster",
>>   "status" : "green",
>>   "timed_out" : false,
>>   "number_of_nodes" : 1,
>>   "number_of_data_nodes" : 1,
>>   "active_primary_shards" : 0,
>>   "active_shards" : 0,
>>   "relocating_shards" : 0,
>>   "initializing_shards" : 0,
>>   "unassigned_shards" : 0
>> }
>>
>> I am on Windows 7 64-bit. 
>> I am using Java 1.7_u55.
>> I am using ES version 1.3.1.
>> I have included in my pom.xml:
>>   - elasticsearch-1.3.1.jar 
>>   - lucene-core-4.9.0.jar
>>
>> Any other suggestions are greatly appreciated.
>>
>>
>>
>> On Tuesday, August 12, 2014 5:45:16 AM UTC-4, Vivek Sachdeva wrote:
>>>
>>> Replace 
>>>
>>> .setTransportAddress(new InetSocketTransportAddress("localhost", 
>>> 9300)); 
>>>
>>> with
>>>
>>> .addTransportAddress(new InetSocketTransportAddress('localhost', 9300)).
>>>
>>> And I guess if you dont give cluster name, it automatically joins the 
>>> default cluster.
>>>
>>> I tried the code that you provided and changed above mentioned code. It 
>>> works on my end. Can you try it?
>>>
>>> On Monday, August 11, 2014 11:34:43 PM UTC+5:30, Kfeenz wrote:

 So I am very new to elasticsearch... so I apologize in advance..

 I started a local instance of elasticsearch and I am trying to connect 
 to it through the Java API.

 I was under the impression that the transport client was for remote 
 clients?

 I tried:

 @Test
 public void testIndexResponse() {

   Client client = new TransportClient().setTransportAddress(new 
 InetSocketTransportAddress("localhost", 9300));

   String json = "{" +
 "\"user\":\"kimchy\"," +

 "\"postDate\":\"2013-01-30\"," +
 "\"message\":\"trying out Elasticsearch\"" +

 "}";

   IndexResponse response = client.prepareIndex("twitter", "tweet")

 .setSource(json)
 .execute()
 .actionGet();


   client.close();

   System.out.println(response.getIndex());
 }

 I receive org.elasticsearch.client.transport.NoNodeAvailableException: 
 None of the configured nodes are available: [].



 On Monday, August 11, 2014 1:19:06 PM UTC-4, Vivek Sachdeva wrote:
>
> Have you tried using transport client for connecting... 
>
> On Monday, August 11, 2014 10:26:29 PM UTC+5:30, Kfeenz wrote:
>>
>> All,
>>
>> I know this post is old, but I continue to have an issue with this... 
>>
>> I get an NoSuchMethodError: org.apache.log4j.Logger.isTraceEnabled()Z 
>> exception when I run
>>
>> Node node = NodeBuilder.nodeBuilder().local(true).node(); // 
>> exception thrown here...
>> Client client = node.client();
>>
>> I have tried including several different slf4j and log4j libraries...
>>
>> Right now I have defined in my pom.xml:
>>
>> slf4j-api-1.7.7.

Re: Embedded ElasticSearch On Java

Actually I am using groovy... So 'localhost' and "localhost" are same fr
me... Are you getting object of transport client in your code...


On Tuesday, August 12, 2014, Kfeenz  wrote:

> @Jorg,
>
> Thanks for the advice, I will make sure that I do so during actual
> implementation, but this is purely for testing the connection.. Also, I see
> a client.close() and a client.threadPool().shutdown(), but I do not see a
> client.threadPool().close(). I am using ES v1.3.1.
>
> @ Vivek,
>
> I am not sure how you were able to use 'localhost' vise "localhost". Java
> complains about an invalid character constant because 'localhost' is not a
> character but a String...
>
> My current code is as follows... with still no luck...
>
> Settings settings = ImmutableSettings.settingsBuilder().put("cluster.name",
> "mycluster").build();
>
> Client client = new TransportClient(settings).addTransportAddress(new
> InetSocketTransportAddress("localhost", 9300));
>
> ClusterStatsRequestBuilder builder =
> client.admin().cluster().prepareClusterStats();
>
> ClusterStatsResponse response = builder.execute().actionGet(); // fails on
> execute... NoNodeAvailableException
>
> assertEquals("mycluster", response.getClusterName()); // never gets to
> this point
>
> NoNodeAvailableException: None of the configured nodes are available []
>
> If I add a setting to the settings object
>
> .put("client.transport.sniff", true);
>
> I get a different error - [org.elasticsearch.client.transport] [Argus]
> failed to get local cluster state info for [#transport#-1]...
>
> I can query the cluster using 
> *http://localhost:9200/_cluster/health?pretty=true
> * which returns
>
> {
>   "cluster_name" : "mycluster",
>   "status" : "green",
>   "timed_out" : false,
>   "number_of_nodes" : 1,
>   "number_of_data_nodes" : 1,
>   "active_primary_shards" : 0,
>   "active_shards" : 0,
>   "relocating_shards" : 0,
>   "initializing_shards" : 0,
>   "unassigned_shards" : 0
> }
>
> I am on Windows 7 64-bit.
> I am using Java 1.7_u55.
> I am using ES version 1.3.1.
> I have included in my pom.xml:
>   - elasticsearch-1.3.1.jar
>   - lucene-core-4.9.0.jar
>
> Any other suggestions are greatly appreciated.
>
>
>
> On Tuesday, August 12, 2014 5:45:16 AM UTC-4, Vivek Sachdeva wrote:
>>
>> Replace
>>
>> .setTransportAddress(new InetSocketTransportAddress("localhost", 9300));
>>
>> with
>>
>> .addTransportAddress(new InetSocketTransportAddress('localhost', 9300)).
>>
>> And I guess if you dont give cluster name, it automatically joins the
>> default cluster.
>>
>> I tried the code that you provided and changed above mentioned code. It
>> works on my end. Can you try it?
>>
>> On Monday, August 11, 2014 11:34:43 PM UTC+5:30, Kfeenz wrote:
>>>
>>> So I am very new to elasticsearch... so I apologize in advance..
>>>
>>> I started a local instance of elasticsearch and I am trying to connect
>>> to it through the Java API.
>>>
>>> I was under the impression that the transport client was for remote
>>> clients?
>>>
>>> I tried:
>>>
>>> @Test
>>> public void testIndexResponse() {
>>>
>>>   Client client = new TransportClient().setTransportAddress(new
>>> InetSocketTransportAddress("localhost", 9300));
>>>
>>>   String json = "{" +
>>> "\"user\":\"kimchy\"," +
>>> "\"postDate\":\"2013-01-30\"," +
>>> "\"message\":\"trying out Elasticsearch\"" +
>>> "}";
>>>
>>>   IndexResponse response = client.prepareIndex("twitter", "tweet")
>>> .setSource(json)
>>> .execute()
>>> .actionGet();
>>>
>>>
>>>   client.close();
>>>
>>>   System.out.println(response.getIndex());
>>> }
>>>
>>> I receive org.elasticsearch.client.transport.NoNodeAvailableException:
>>> None of the configured nodes are available: [].
>>>
>>>
>>>
>>> On Monday, August 11, 2014 1:19:06 PM UTC-4, Vivek Sachdeva wrote:

 Have you tried using transport client for connecting...

 On Monday, August 11, 2014 10:26:29 PM UTC+5:30, Kfeenz wrote:
>
> All,
>
> I know this post is old, but I continue to have an issue with this...
>
> I get an NoSuchMethodError: org.apache.log4j.Logger.isTraceEnabled()Z
> exception when I run
>
> Node node = NodeBuilder.nodeBuilder().local(true).node(); //
> exception thrown here...
> Client client = node.client();
>
> I have tried including several different slf4j and log4j libraries...
>
> Right now I have defined in my pom.xml:
>
> slf4j-api-1.7.7.jar
> slf4j-ext-1.7.7.jar
> log4j-core-2.0.jar
> log4j-api-2.0.jar
>
> Any suggestions?
> log4j-slf4j-impl-2.0.jar
>
> On Wednesday, November 30, 2011 6:55:57 PM UTC-5, Sam wrote:
>>
>> All
>>
>> I have a webapp (JAVA) with a list of items on the page. All these
>> items are generated on html with session variables. I want to use
>> ElasticSearch on the back end to do a search for items with different
>> search criteria. Is th

Re: Can't delete Index!

Just wondering, will remove/re-install get rid of the stagnant index since 
I plan on upgrading from 1.2.1 to 1.2.2?

On Tuesday, August 12, 2014 9:42:10 AM UTC-4, Sam2014 wrote:
>
> The link doesn't work.
> I am not sure about the difference, I have always deleted the indices the 
> same way, curl -XPUT http... //to create, and curl -XDelete /http:...
> I did not create any aliases, I am just using Get _aliases to see whats 
> still on the node.
> Did the log tell you anything, there are some weird errors in there that I 
> don't know how to interpret, a quick look there may tell you something.
>
> On Tuesday, August 12, 2014 3:04:39 AM UTC-4, Jörg Prante wrote:
>>
>> So you are wrestling with aliases. You can not delete aliases by file 
>> system operations. Have you checked
>>
>>
>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-aliases.html#deleting
>>
>> for deleting aliases?
>>
>> Jörg
>>
>>
>> On Tue, Aug 12, 2014 at 4:10 AM, Sam2014  wrote:
>>
>>> ///Attached is the cluster log that covers the whole sequence from 
>>> shutting the cluster and re-activating it.
>>>
>>> //Show me current state of cluster
>>> {
>>>"cluster_name": "rexCluster",
>>>"status": "red",
>>>"timed_out": false,
>>>"number_of_nodes": 2,
>>>"number_of_data_nodes": 2,
>>>"active_primary_shards": 7,
>>>"active_shards": 7,
>>>"relocating_shards": 0,
>>>"initializing_shards": 2,
>>>"unassigned_shards": 21
>>> }
>>>
>>> //Show me indices on the cluster
>>>
>>> GET _aliases
>>> {
>>>"foo": {
>>>   "aliases": {}
>>>},
>>>"mediregs": {
>>>   "aliases": {}
>>>},
>>>"twitter": {
>>>   "aliases": {}
>>>}
>>> }
>>>
>>>
>>>
>>> //Shut down the cluster
>>>
>>> POST /_cluster/nodes/_shutdown
>>> {
>>>"cluster_name": "rexCluster",
>>>"nodes": {
>>>   "_vPEfmc9TruIpugk4NWleg": {
>>>  "name": "node 1"
>>>   },
>>>   "DB94rWiiSvS9HUTXwI7GTw": {
>>>  "name": "node 2"
>>>   }
>>>}
>>> }
>>>
>>>  //Go into the fs and delete files under /data
>>>
>>> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0$
>>>  
>>> ls
>>>
>>> *indices*  node.lock  *_state*
>>>
>>> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0$
>>>  
>>> cd indices/
>>>
>>> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$
>>>  
>>> ls
>>>
>>> *foo*  *mediregs*  *twitter*
>>>
>>> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$
>>>  
>>> sudo rm -rf foo
>>>
>>> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$
>>>  
>>> sudo rm -rf mediregs
>>>
>>> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$
>>>  
>>> sudo rm -rf twitter
>>>
>>> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$
>>>  
>>> ls
>>>
>>>
>>> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$
>>>  
>>> //Restart nodes 1 &2 - should be in log file
>>>
>>> Get _aliases //Again - they came back!!
>>> {
>>>"foo": {
>>>   "aliases": {}
>>>},
>>>"mediregs": {
>>>   "aliases": {}
>>>},
>>>"twitter": {
>>>   "aliases": {}
>>>}
>>> }
>>>
>>>
>>> //Ok - Try to delete index
>>> DELETE /mediregs
>>>
>>> //takes little while and comes back
>>> {
>>>"acknowledged": false
>>> }
>>>
>>>
>>>
>>>
>>> On Monday, August 11, 2014 7:12:22 PM UTC-4, Jörg Prante wrote:
>>>
 Can you post your detailed steps and debug logs?

 Make also sure there is no third node - this looks like ES is shipping 
 shards to a third node from which the two nodes are recovering from.

 Jörg


 On Tue, Aug 12, 2014 at 1:06 AM, Sam2014  wrote:

> I stopped node 1, then I stopped node 2. I disabled the cluster, 
> deleted files from nodes, Made sure no Indices on either node. I 
> re-enabled 
> cluster in yml, restarted cluster, and the index came back!
>
>
> On Monday, August 11, 2014 6:16:42 PM UTC-4, Jörg Prante wrote:
>
>> There is no vicious circle. First you must stop the cluster. Then, 
>> after processes have exited, continue with file operations. After that, 
>> you 
>> can start the cluster nodes again.
>>
>> Jörg
>>
>>
>> On Mon, Aug 11, 2014 at 11:49 PM, Sam2014  wrote:
>>
>>> Oops, When I restarted the cluster/nodes, I see shards under an 
>>> "unassigned node", neither of the two nodes of the cluster. Is there a 
>>> way 
>>> I can route/assign /delete out of this vicious cycle?
>>>
>>>
>>> On Monday, August 11, 2014 5:46:25 PM UTC-4, Sam2014 wrote:

 I also deleted the directory, restarted the nodes, and it recovered 
 the index still, which is null.

 On Monday, August 11, 2014 5:42:53 PM UTC-4, Sam2014 wrot

Re: Why my buckets's doc_count is limit to 3000?

2014-08-12 Thread humen1


indexing speed.

在 2014年8月12日，21:27，Adrien Grand  写道：

> I am not sure to understand if your question is about aggregations or 
> indexing speed?
> 
> 
> On Tue, Aug 12, 2014 at 11:24 AM, 陈浩  wrote:
> this is my DSL
> 
> {
> "query":{
> "match_all":{}
> },
>
> "aggs": {
> "range": {
> "date_range": {
> "field": "@timestamp", "format": "yyy.MM.dd.HH.mm.ss",
> "ranges": [{"from": "2014.08.12.09.18.45", "to": 
> "2014.08.12.09.20.50"}]
> },  
> "aggs": {
> "over_time": {
> "date_histogram": {
> "field": "@timestamp",
> "interval": "1s",
> "format": "yyy.MM.dd.HH.mm.ss"
> },
> "aggs": {
> "total_sent": {
> "sum": {"field": "bytes_sent"}
> }
> }
> }
> }
> }
> }
> }
> 
> 
>  
> 
> thus is my result
> 
> {
> 
> "took":6,
> 
> "timed_out":false,
> "_shards":{
> 
> "total":96,
> 
> "successful":96,
> "failed":0
> 
> },
> "hits":{
> 
> "total":258002,
> 
> "max_score":0.0,
> "hits":[
> 
> ]
> },
> "aggregations":{
> 
> "range":{
> 
> "buckets":[
> 
> {
> 
> "key":"2014.08.12.09.18.45-2014.08.12.09.20.50",
> 
> "from":1.407835125E12,
> "from_as_string":"2014.08.12.09.18.45",
> 
> "to":1.40783525E12,
> 
> "to_as_string":"2014.08.12.09.20.50",
> 
> "doc_count":12000,
> "over_time":{
> 
> "buckets":[
> 
> {
> 
> "key_as_string":"2014.08.12.09.18.45",
> 
> "key":1407835125000,
> "doc_count":3000,
> 
> "total_sent":{
> 
> "value":6.6126308E7
> }
> 
> },
> {
> 
> "key_as_string":"2014.08.12.09.18.47",
> 
> "key":1407835127000,
> "doc_count":3000,
> 
> "total_sent":{
> 
> "value":9.286586E7
> }
> 
> },
> {
> 
> "key_as_string":"2014.08.12.09.18.49",
> 
> "key":1407835129000,
> "doc_count":3000,
> 
> "total_sent":{
> 
> "value":1.21316184E8
> }
> 
> },
> {
> 
> "key_as_string":"2014.08.12.09.18.51",
> 
> "key":1407835131000,
> "doc_count":3000,
> 
> "total_sent":{
> 
> "value":8.3529544E7
> }
> 
> }
> ]
> }
> }
> ]
> 
> }
> }
> }
> 
> 
> 
> 
> 
> 
> my soft flow is:
> Nginx ==pipe==> syslogng ==udp==> logstash ==es_river==> elasticsearch
> 
> How can I break the 3000doc/sec ?
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/1fa43dda-31cd-4f10-95bc-0f31778d9663%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
> 
> 
> 
> -- 
> Adrien Grand
> 
> -- 
> You received this message because you are subscribed to a topic in the Google 
> Groups "elasticsearch" group.
> To unsubscribe from this topic, visit 
> https://groups.google.com/d/topic/elasticsearch/qj4BP7rEnTU/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to 
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6kYQAU6ExwMFwxTuy3LQQx%2BLSh1_-wNO7cxTd9GbF-Ow%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/EFF0A61B-AB24-4C8B-ABE2-A1B2D5FCC5AC%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Uneven Primary Shard Distribution in Cluster

2014-08-12 Thread Andrew Ruslander

I have a five node Elasticsearch cluster set up so that two nodes are in 
one zone and the other three nodes are in a different zone (let's call them 
SideA and SideB) via use of the forced awareness attributes.  I also have a 
sixth node that has Logstash on it.  Logstash is outputting to one of the 
two nodes in SideA.  However, when I use Marvel to view the shard 
allocation across my five node cluster, I see that probably 95% of the 
primary shards are all on SideB.  SideA is almost exclusively replicas. 
 Just out of curiosity, I changed Logstash to output to a node in SideB, 
but the primaries continued to be allocated to SideB only.

So my question is...is this expected behavior?  What would cause the 
primary shards to be allocated only to one side?  Is it because it has 
three nodes versus the two nodes on SideA?  Is something else afoot here?

- Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/dba612d0-adb9-4bca-8c67-3b35e9236bc7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: clarity for shard allocation disable/enable during upgrade

2014-08-12 Thread bitsofinfo . g

Also, Clinton, per the upgrade page it states the below, so what you are 
saying is that re-enabling allocation after each node is restarted (going 
from 1.2.1 to 1.31) that the below *will not* apply (incompatibility) 
because shards would be going from 1.2.1 to 1.3.1 vs the reverse Correct?

"Running multiple versions of Elasticsearch in the same cluster for any 
length of time beyond that required for an upgrade is not supported, as 
shard replication from the more recent version to the previous versions 
will not work."

On Tuesday, August 12, 2014 4:04:28 AM UTC-4, Clinton Gormley wrote:
>
>
>
> On Monday, 11 August 2014 15:31:28 UTC+2, bitsof...@gmail.com wrote:
>>
>> I have 8 data nodes and 6 coordinator nodes in an active cluster running 
>> 1.2.1
>>
>> I want to upgrade to 1.3.1
>>
>> When reading 
>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html
>>  
>> the upgrade docs am I correct to assume:
>>
>> a) disable shard allocation before doing anything
>>
>> b) proceed to upgrade each node to 1.3.1
>>
>> c) only after ALL nodes are @ 1.3.1 then I can re-enable shard allocation.
>>
>> My question is that at some point during the upgrade of all the data 
>> nodes, the shards on them will be "unassigned" and the cluster will not 
>> function... correct?
>>
>> So in other words running some nodes as 1.2.1 and others as 1.3.1 with 
>> shard allocation *enabled* is NOT advised and in general cluster 
>> un-availability is expected due to shards being in an unassigned state as 
>> each data node is upgraded.
>>
>> At least this is the behavior I see today, (not during an upgrade) when I 
>> disable allocation and restart a node, those shards are unassigned until I 
>> re-enable allocation
>>
>>
> No, the procedure outlined above is not correct and would indeed result in 
> unassigned shards, as you suspect.  Instead, you should:
>
> 1. Disable allocation
> 2. Upgrade ONE node
> 3. Reenable allocation
> 4. Wait for green
> 5. Repeat
>
> Even when following the above process, you will likely end up with shards 
> being copied over from one node to another (once allocation has been 
> reenabled).  After restart, a replica will only reuse the segments that are 
> exactly the same as those in the primary.  However, because primaries and 
> replicas refresh, flush, and merge at different times, shards diverge from 
> each other over time. The longer it has been since a replica was copied 
> over from the primary, the fewer identical segments they will have in 
> common.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/0dc5d597-3658-447f-94e0-b4b357fa4b08%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: clarity for shard allocation disable/enable during upgrade

2014-08-12 Thread bitsofinfo . g

Mark - isn't the shard allocation all/none a cluster wide setting? Hence 
why it does that on all nodes?

Clinton - What you said makes sense, however if that procedure is incorrect 
then the official upgrade page on the elasticsearch site should be changed, 
as it states

"When the process is complete on all nodes, you can re-enable shard 
reallocation"


On Tuesday, August 12, 2014 4:04:28 AM UTC-4, Clinton Gormley wrote:
>
>
>
> On Monday, 11 August 2014 15:31:28 UTC+2, bitsof...@gmail.com wrote:
>>
>> I have 8 data nodes and 6 coordinator nodes in an active cluster running 
>> 1.2.1
>>
>> I want to upgrade to 1.3.1
>>
>> When reading 
>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html
>>  
>> the upgrade docs am I correct to assume:
>>
>> a) disable shard allocation before doing anything
>>
>> b) proceed to upgrade each node to 1.3.1
>>
>> c) only after ALL nodes are @ 1.3.1 then I can re-enable shard allocation.
>>
>> My question is that at some point during the upgrade of all the data 
>> nodes, the shards on them will be "unassigned" and the cluster will not 
>> function... correct?
>>
>> So in other words running some nodes as 1.2.1 and others as 1.3.1 with 
>> shard allocation *enabled* is NOT advised and in general cluster 
>> un-availability is expected due to shards being in an unassigned state as 
>> each data node is upgraded.
>>
>> At least this is the behavior I see today, (not during an upgrade) when I 
>> disable allocation and restart a node, those shards are unassigned until I 
>> re-enable allocation
>>
>>
> No, the procedure outlined above is not correct and would indeed result in 
> unassigned shards, as you suspect.  Instead, you should:
>
> 1. Disable allocation
> 2. Upgrade ONE node
> 3. Reenable allocation
> 4. Wait for green
> 5. Repeat
>
> Even when following the above process, you will likely end up with shards 
> being copied over from one node to another (once allocation has been 
> reenabled).  After restart, a replica will only reuse the segments that are 
> exactly the same as those in the primary.  However, because primaries and 
> replicas refresh, flush, and merge at different times, shards diverge from 
> each other over time. The longer it has been since a replica was copied 
> over from the primary, the fewer identical segments they will have in 
> common.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/095cf279-5c51-49bc-8fb3-536e3160a9ec%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Embedded ElasticSearch On Java

2014-08-12 Thread Kfeenz

@Jorg,

Thanks for the advice, I will make sure that I do so during actual 
implementation, but this is purely for testing the connection.. Also, I see 
a client.close() and a client.threadPool().shutdown(), but I do not see a 
client.threadPool().close(). I am using ES v1.3.1.

@ Vivek,

I am not sure how you were able to use 'localhost' vise "localhost". Java 
complains about an invalid character constant because 'localhost' is not a 
character but a String...

My current code is as follows... with still no luck...

Settings settings = ImmutableSettings.settingsBuilder().put("cluster.name", 
"mycluster").build();

Client client = new TransportClient(settings).addTransportAddress(new 
InetSocketTransportAddress("localhost", 9300));

ClusterStatsRequestBuilder builder = 
client.admin().cluster().prepareClusterStats();

ClusterStatsResponse response = builder.execute().actionGet(); // fails on 
execute... NoNodeAvailableException

assertEquals("mycluster", response.getClusterName()); // never gets to this 
point

NoNodeAvailableException: None of the configured nodes are available []

If I add a setting to the settings object

.put("client.transport.sniff", true);

I get a different error - [org.elasticsearch.client.transport] [Argus] 
failed to get local cluster state info for [#transport#-1]...

I can query the cluster using 
*http://localhost:9200/_cluster/health?pretty=true* which returns

{
  "cluster_name" : "mycluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0
}

I am on Windows 7 64-bit. 
I am using Java 1.7_u55.
I am using ES version 1.3.1.
I have included in my pom.xml:
  - elasticsearch-1.3.1.jar 
  - lucene-core-4.9.0.jar

Any other suggestions are greatly appreciated.



On Tuesday, August 12, 2014 5:45:16 AM UTC-4, Vivek Sachdeva wrote:
>
> Replace 
>
> .setTransportAddress(new InetSocketTransportAddress("localhost", 9300)); 
>
> with
>
> .addTransportAddress(new InetSocketTransportAddress('localhost', 9300)).
>
> And I guess if you dont give cluster name, it automatically joins the 
> default cluster.
>
> I tried the code that you provided and changed above mentioned code. It 
> works on my end. Can you try it?
>
> On Monday, August 11, 2014 11:34:43 PM UTC+5:30, Kfeenz wrote:
>>
>> So I am very new to elasticsearch... so I apologize in advance..
>>
>> I started a local instance of elasticsearch and I am trying to connect to 
>> it through the Java API.
>>
>> I was under the impression that the transport client was for remote 
>> clients?
>>
>> I tried:
>>
>> @Test
>> public void testIndexResponse() {
>>
>>   Client client = new TransportClient().setTransportAddress(new 
>> InetSocketTransportAddress("localhost", 9300));
>>
>>   String json = "{" +
>> "\"user\":\"kimchy\"," +
>> "\"postDate\":\"2013-01-30\"," +
>> "\"message\":\"trying out Elasticsearch\"" +
>> "}";
>>
>>   IndexResponse response = client.prepareIndex("twitter", "tweet")
>> .setSource(json)
>> .execute()
>> .actionGet();
>>
>>
>>   client.close();
>>
>>   System.out.println(response.getIndex());
>> }
>>
>> I receive org.elasticsearch.client.transport.NoNodeAvailableException: 
>> None of the configured nodes are available: [].
>>
>>
>>
>> On Monday, August 11, 2014 1:19:06 PM UTC-4, Vivek Sachdeva wrote:
>>>
>>> Have you tried using transport client for connecting... 
>>>
>>> On Monday, August 11, 2014 10:26:29 PM UTC+5:30, Kfeenz wrote:

 All,

 I know this post is old, but I continue to have an issue with this... 

 I get an NoSuchMethodError: org.apache.log4j.Logger.isTraceEnabled()Z 
 exception when I run

 Node node = NodeBuilder.nodeBuilder().local(true).node(); // exception 
 thrown here...
 Client client = node.client();

 I have tried including several different slf4j and log4j libraries...

 Right now I have defined in my pom.xml:

 slf4j-api-1.7.7.jar
 slf4j-ext-1.7.7.jar
 log4j-core-2.0.jar
 log4j-api-2.0.jar

 Any suggestions?
 log4j-slf4j-impl-2.0.jar

 On Wednesday, November 30, 2011 6:55:57 PM UTC-5, Sam wrote:
>
> All
>
> I have a webapp (JAVA) with a list of items on the page. All these
> items are generated on html with session variables. I want to use
> ElasticSearch on the back end to do a search for items with different
> search criteria. Is there a sample I can use it as a resource?
>
> Thanks
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/090eaed6-ce13-4460-8db1-b69

Re: Elasticsearch with 1.8 JRE?

2014-08-12 Thread Derry O' Sullivan

Hi Jorg,

Thanks for the fast response. Looks like it affects jre7U65 so i guess the
safe options are:
1.7.0 (55/60) or 1.8.0 (5)

Derry


On 12 August 2014 15:09, joergpra...@gmail.com 
wrote:

> You can use Java 8u5
>
> Do not use Java 8u11 with Groovy or Guava, there is a bug
> https://bugs.openjdk.java.net/browse/JDK-8051012
>
> Jörg
>
>
> On Tue, Aug 12, 2014 at 4:00 PM, Derry O' Sullivan 
> wrote:
>
>> Hi all,
>>
>> I know that JRE 1.7.0_55 is the 'recommended' jre for ES 1.2+:
>> http://www.elasticsearch.org/blog/java-1-7u55-safe-use-elasticsearch-lucene/
>>
>> There seems to have been some posts regarding Java 8/MVEL issues (e.g.
>> http://jira.codehaus.org/browse/MVEL-299) but is the current JRE 8 (11)
>> ok for production ES usage?
>>
>> Didn't spot anything on the lucene side of things so i guess more of an
>> ES compilation/test question.
>> http://wiki.apache.org/lucene-java/JavaBugs
>>
>> Thanks,
>>
>> Derry
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>>
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/cd344c47-0f6e-4254-b0fe-5d2f03bb3797%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/NbvgObj_y2s/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGFb%2Be%2BgyZTjU9hyCuDhO60q2332R2yRQ1%3DUG63EmJ1Tg%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAOGQnJrL1NUyUqBnRPCoV_mvm6sk4xz8BottDZhSDTS4WmBAyw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch with 1.8 JRE?

You can use Java 8u5

Do not use Java 8u11 with Groovy or Guava, there is a bug
https://bugs.openjdk.java.net/browse/JDK-8051012

Jörg


On Tue, Aug 12, 2014 at 4:00 PM, Derry O' Sullivan 
wrote:

> Hi all,
>
> I know that JRE 1.7.0_55 is the 'recommended' jre for ES 1.2+:
> http://www.elasticsearch.org/blog/java-1-7u55-safe-use-elasticsearch-lucene/
>
> There seems to have been some posts regarding Java 8/MVEL issues (e.g.
> http://jira.codehaus.org/browse/MVEL-299) but is the current JRE 8 (11)
> ok for production ES usage?
>
> Didn't spot anything on the lucene side of things so i guess more of an ES
> compilation/test question.
> http://wiki.apache.org/lucene-java/JavaBugs
>
> Thanks,
>
> Derry
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/cd344c47-0f6e-4254-b0fe-5d2f03bb3797%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGFb%2Be%2BgyZTjU9hyCuDhO60q2332R2yRQ1%3DUG63EmJ1Tg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Elasticsearch with 1.8 JRE?

2014-08-12 Thread Derry O' Sullivan

Hi all,

I know that JRE 1.7.0_55 is the 'recommended' jre for ES 
1.2+:http://www.elasticsearch.org/blog/java-1-7u55-safe-use-elasticsearch-lucene/

There seems to have been some posts regarding Java 8/MVEL issues 
(e.g. http://jira.codehaus.org/browse/MVEL-299) but is the current JRE 8 
(11) ok for production ES usage?

Didn't spot anything on the lucene side of things so i guess more of an ES 
compilation/test question.
http://wiki.apache.org/lucene-java/JavaBugs

Thanks,

Derry

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/cd344c47-0f6e-4254-b0fe-5d2f03bb3797%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Custom Aggregations

Hi Jin,

It is possible to write a custom aggregation, you can for example look at
this plugin: https://github.com/algolia/elasticsearch-cardinality-plugin
that implements a aggregation as a plugin.

If that would work for you, another option would be to contribute this
aggregation to Elaticsearch so that it would be available out of the box
(feel free to open an issue on Github and ping me (@jpountz), I'd be happy
to help).


On Sat, Aug 9, 2014 at 8:19 PM, Jin Huang  wrote:

> Hi Alex,
>
> Is there an updates on this topic? I want to know how easy or how complex
> is it to write a custom aggregator. For example, I want to calculate all
> the n-th central moments, with n=2,3,4,..10. By the way, Elasticsearch
> already provides the 2nd central moment, i.e. square of the standard
> deviation.
>
> In general, I would like to know what kind of custom aggregator is easy to
> write?
>
> Thanks,
> Jin
>
>
> On Wednesday, February 12, 2014 5:49:28 AM UTC-8, Alexander Reelsen wrote:
>
>> Hey,
>>
>> sure, you can write a custom plugin, which implements your custom
>> aggregations, if you want to.
>> Any specific functionality you are missing (might be interesting to get
>> it in the core anyway?).
>>
>>
>> --Alex
>>
>>
>> On Tue, Feb 11, 2014 at 9:41 PM, Justin Uang  wrote:
>>
>>> Is there any way we can define our own aggregation functions beyond the
>>> provided metric and bucket aggregations?
>>>
>>> Thanks!
>>>
>>> Justin
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>>
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/elasticsearch/90accf78-6540-4126-8268-7f997353ad74%
>>> 40googlegroups.com.
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/b2b8894d-c26d-4787-af4c-c45f793befc6%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Adrien Grand

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5pEbWQKAptt10jtwOwuLRXJ5SzLmDY2yn5f_4eqeaY1Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Can't delete Index!

The link doesn't work.
I am not sure about the difference, I have always deleted the indices the 
same way, curl -XPUT http... //to create, and curl -XDelete /http:...
I did not create any aliases, I am just using Get _aliases to see whats 
still on the node.
Did the log tell you anything, there are some weird errors in there that I 
don't know how to interpret, a quick look there may tell you something.

On Tuesday, August 12, 2014 3:04:39 AM UTC-4, Jörg Prante wrote:
>
> So you are wrestling with aliases. You can not delete aliases by file 
> system operations. Have you checked
>
>
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-aliases.html#deleting
>
> for deleting aliases?
>
> Jörg
>
>
> On Tue, Aug 12, 2014 at 4:10 AM, Sam2014 
> > wrote:
>
>> ///Attached is the cluster log that covers the whole sequence from 
>> shutting the cluster and re-activating it.
>>
>> //Show me current state of cluster
>> {
>>"cluster_name": "rexCluster",
>>"status": "red",
>>"timed_out": false,
>>"number_of_nodes": 2,
>>"number_of_data_nodes": 2,
>>"active_primary_shards": 7,
>>"active_shards": 7,
>>"relocating_shards": 0,
>>"initializing_shards": 2,
>>"unassigned_shards": 21
>> }
>>
>> //Show me indices on the cluster
>>
>> GET _aliases
>> {
>>"foo": {
>>   "aliases": {}
>>},
>>"mediregs": {
>>   "aliases": {}
>>},
>>"twitter": {
>>   "aliases": {}
>>}
>> }
>>
>>
>>
>> //Shut down the cluster
>>
>> POST /_cluster/nodes/_shutdown
>> {
>>"cluster_name": "rexCluster",
>>"nodes": {
>>   "_vPEfmc9TruIpugk4NWleg": {
>>  "name": "node 1"
>>   },
>>   "DB94rWiiSvS9HUTXwI7GTw": {
>>  "name": "node 2"
>>   }
>>}
>> }
>>
>>  //Go into the fs and delete files under /data
>>
>> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0$
>>  
>> ls
>>
>> *indices*  node.lock  *_state*
>>
>> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0$
>>  
>> cd indices/
>>
>> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$
>>  
>> ls
>>
>> *foo*  *mediregs*  *twitter*
>>
>> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$
>>  
>> sudo rm -rf foo
>>
>> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$
>>  
>> sudo rm -rf mediregs
>>
>> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$
>>  
>> sudo rm -rf twitter
>>
>> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$
>>  
>> ls
>>
>>
>> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$
>>  
>> //Restart nodes 1 &2 - should be in log file
>>
>> Get _aliases //Again - they came back!!
>> {
>>"foo": {
>>   "aliases": {}
>>},
>>"mediregs": {
>>   "aliases": {}
>>},
>>"twitter": {
>>   "aliases": {}
>>}
>> }
>>
>>
>> //Ok - Try to delete index
>> DELETE /mediregs
>>
>> //takes little while and comes back
>> {
>>"acknowledged": false
>> }
>>
>>
>>
>>
>> On Monday, August 11, 2014 7:12:22 PM UTC-4, Jörg Prante wrote:
>>
>>> Can you post your detailed steps and debug logs?
>>>
>>> Make also sure there is no third node - this looks like ES is shipping 
>>> shards to a third node from which the two nodes are recovering from.
>>>
>>> Jörg
>>>
>>>
>>> On Tue, Aug 12, 2014 at 1:06 AM, Sam2014  wrote:
>>>
 I stopped node 1, then I stopped node 2. I disabled the cluster, 
 deleted files from nodes, Made sure no Indices on either node. I 
 re-enabled 
 cluster in yml, restarted cluster, and the index came back!


 On Monday, August 11, 2014 6:16:42 PM UTC-4, Jörg Prante wrote:

> There is no vicious circle. First you must stop the cluster. Then, 
> after processes have exited, continue with file operations. After that, 
> you 
> can start the cluster nodes again.
>
> Jörg
>
>
> On Mon, Aug 11, 2014 at 11:49 PM, Sam2014  wrote:
>
>> Oops, When I restarted the cluster/nodes, I see shards under an 
>> "unassigned node", neither of the two nodes of the cluster. Is there a 
>> way 
>> I can route/assign /delete out of this vicious cycle?
>>
>>
>> On Monday, August 11, 2014 5:46:25 PM UTC-4, Sam2014 wrote:
>>>
>>> I also deleted the directory, restarted the nodes, and it recovered 
>>> the index still, which is null.
>>>
>>> On Monday, August 11, 2014 5:42:53 PM UTC-4, Sam2014 wrote:

 I wish it was that simple, I deleted the directory of the index 
 from /data. Yet, the index is still there and won't be deleted or 
 recreated.. I still get acknowledged = false.

 On Monday, August 11, 2014 5:31:36 PM UTC-4, Jörg Prante wrote:
>
> Your JVMs (at least one of them) are not abl

Re: Paging for nested objects.

Hi,

Unfortunately this is not possible.


On Mon, Aug 11, 2014 at 12:48 PM,  wrote:

> Hello, I'm new in elasticsearch and I have a question.
> How can I do paging for nested object and is it possible? (I need to get
> comments with paging)
>
> *My mappping:*
> {
>"test": {
>   "mappings": {
>  "completion": {
> "properties": {
>"suggest": {
>   "type": "string",
>   "index": "not_analyzed"
>}
> }
>  },
>  "post": {
> "properties": {
>"comments": {
>
>   "type": "nested",
>   "properties": {
>  "created_time": {
> "type": "date",
> "format": "dateOptionalTime"
>  },
>  "id": {
> "type": "string",
> "index": "not_analyzed"
>  },
>  "text": {
> "type": "string"
>  }
>   }
>},
>"created_time": {
>   "type": "date",
>   "format": "dateOptionalTime"
>},
>"text": {
>   "type": "string"
>},
>"thumbnail": {
>   "type": "string",
>   "index": "not_analyzed"
>},
>"title": {
>   "type": "string"
>}
> }
>  }
>   }
>}
> }
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/23fa5330-be0d-410c-99db-8eb924c0bdac%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Adrien Grand

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5JfbQqet3Z_dJY%3D6n0J0stLFky7LMmvrsxyPekeciGdQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Creating filters per aggregation similar to Facets

2014-08-12 Thread Ivan Brusic

Trying using a filter aggregation:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-filter-aggregation.html

The idea is that the filter is the outer most aggregation and the
aggregation you actually want to filter is the sub-aggregation.

Cheers,

Ivan


On Mon, Aug 11, 2014 at 6:18 PM, Jeff Steinmetz  wrote:

> Kibana provides a good example of date histograms, split out by each
> "query" entered at the top in the "Query" bar.  It essentially creates
> multiple free text queries against "all".
>
> I see it generates per facet filter, with a free text (query_string)
> search.
> Since facets are to be depreciated, I am now only using aggregations (in a
> custom application - unrelated to Kibana).  I have tried this with
> aggregations without success.
> I also realize there is something new coming in 1.4, but I assume with
> multiple aggregations, (vs. multiple filters to create multiple buckets) I
> can do this today.
>
> Here is a oversimplified version of the date histogram aggregation I have
> (without the leading query section - consider it pseudo code)
>
> The "filter" section is the part in question.  Removing the filter works,
> I have tried all types of "filter" formats, looked for samples, etc. no
> luck.  I have tried {"all" : "search term"}
> as well as:
> {"query_string": { "all" : "search terrm" }}
>
> I've tried a specific field name, etc.  All attempts are not proving
> fruitful.
>
> Pseudo example using aggregations:
> 
>
>   "aggregations" : {
> "0" : {
>   "date_histogram" : {
>   "filter" : { "query_string" : { "query" : "Intel" } },
> "field" : "created_at",
> "interval" : "1d",
> "min_doc_count" : 0
>   }
> },
>"1" : {
>   "date_histogram" : {
>   "filter" : { "query_string" : { "query" : "Samsung" } },
> "field" : "created_at",
> "interval" : "1d",
> "min_doc_count" : 0,
> "pre_zone" : "-02:00",
> "post_zone" : "-03:30"
>   }
> }
>   }
>
>
> Here is the Facet version (which works - note
> Filtered/query/query_string/query):
> ---
>
> {
> "facets": {
> "0": {
> "date_histogram": {
> "field": "created_at",
> "interval": "3h"
> },
> "global": true,
> "facet_filter": {
> "fquery": {
> "query": {
> "filtered": {
> "query": {
> "query_string": {
> "query": "Intel"
> }
> },
> "filter": {
> "bool": {
> "must": [
> {
> "terms": {
> "userid": [
>
> "53d02d6aed9597f3c6fa"
> ]
> }
> },
> {
> "range": {
> "created_at": {
> "from": "now-30d",
> "to": "now"
> }
> }
> }
> ]
> }
> }
> }
> }
> }
> }
> },
> "1": {
> "date_histogram": {
> "field": "created_at",
> "interval": "3h"
> },
> "global": true,
> "facet_filter": {
> "fquery": {
> "query": {
> "filtered": {
> "query": {
> "query_string": {
> "query": "Samsung"
> }
> },
> "filter": {
> "bool": {
> "must": [
> {
> "terms": {
> "userid": [
>
> "53d02d6aed9597f3c6fa"
> ]
> }
> },
> {
> "range": {
>

Re: nested fields and _field_names

_field_names tracks the field names of the current index document, so if
you need to be in the context of your nested documents to aggregate on
their field names. That would give something like:

GET test/_search
{
  "aggs": {
"nested_docs": {
  "nested": {
"path": "a"
  },
  "aggs": {
"top_fields": {
  "terms": {
"field": "_field_names",
"size": 10
  }
}
  }
}
  }
}


On Tue, Aug 12, 2014 at 5:49 AM, Peter Li  wrote:

> I am trying to profile fields by counting documents with each given field.
> I am using 1.3.0 ES, so I thought I can use:
>
> aggs: {
>docs : {
>   terms : {
>  field : _field_names,
>  size : 0
>   }
>}
> }
>
> to give a count of documents with each specific field/path.
>
> But this returns nothing for nested fields. When I use a mapping of
>
> a : {
>type : nested,
>properties: {
>   x : {...}
>   y : {...}
>}
> }
>
> for document structures like
>
> a : [
>   {  x : ..., y: ... },
>   {  x : ..., y: ... }
> ]
>
> The aggregate returns nothing for x and y. Is this a bug ?
>
> Thanks,
> Peter Li
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/c535ad64-ed80-446b-9927-13a424556a1c%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Adrien Grand

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6NcZrxezb%2BnEOcOQY-BCqazF3tp9oNGB130_fh_SooGw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Segfault in ffi_prep_closure_loc

There are so many VMs but, together with native code paths like JNA +
mlockall + libffi etc. the chance that something odd breaks down the road
is always there...

With Vmware VM and mlockall, not everything can work like on bare metal.
Memory management strongly depends on the host OS settings. The guest can
not control the host resource management, memory pinning, swap etc. Beside
this: bug is bug, libffi should not break with SIGSEGV, in any case.

Jörg



On Tue, Aug 12, 2014 at 3:29 PM, Markus Burger 
wrote:

> Yeah its a Vmware VM but i have a couple of ES 1.1 sitting in another
> Cluster without Issues, are there known issues with running ES in a VM ?
>
> Thanks Jörg! I'll update libffi the next couple of days
>
> markus
>
> Am Dienstag, 12. August 2014 15:12:16 UTC+2 schrieb Jörg Prante:
>>
>> Same configuration runs here, on bare metal. Maybe you run a VM?
>>
>> Just a shot in the dark, it looks like a bug in libffi, and there are
>> more recent versions like 3.0.5, so if you feel like hacking this bug, I
>> would try if libffi 3.1 from https://github.com/atgreen/libffi/ gives a
>> different picture.
>>
>> Jörg
>>
>>
>> On Tue, Aug 12, 2014 at 2:59 PM, Markus Burger 
>> wrote:
>>
>>> exactly...
>>>
>>> [root@dev ~]# rpm -q libffi
>>> libffi-3.0.5-3.2.el6.x86_64
>>>
>>> markus
>>>
>>> Am Dienstag, 12. August 2014 14:51:45 UTC+2 schrieb Jörg Prante:

 And

 rpm -q libffi

 gives

 libffi-3.0.5-3.2.el6.x86_64

 ?

 Jörg



 On Tue, Aug 12, 2014 at 2:19 PM, Markus Burger 
 wrote:

> [root@dev limits.d]# runuser -s /bin/bash elasticsearch -c 'ulimit -l'
> unlimited
> [root@dev limits.d]# sysctl vm.max_map_count
> vm.max_map_count = 262144
>
> markus
>
> Am Dienstag, 12. August 2014 13:58:30 UTC+2 schrieb Jörg Prante:
>
>> "S User" --> "ES user"
>>
>>  --
> You received this message because you are subscribed to the Google
> Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to elasticsearc...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/elasticsearch/044b2715-c566-476b-ba8f-035cb4cf0af6%40goo
> glegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

  --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/elasticsearch/8485ac49-7d23-47f2-b28f-0b475052d293%
>>> 40googlegroups.com
>>> 
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/b6e894da-6ebe-4f30-9a7b-7daf76b4b4d4%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFz8H38HgFp-5ssBTebgey2Jxg%3DB4R533qOjrMTB6paew%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: impact of stored fields on performance

Hi Ashish,

How many documents do your queries typically retrieve? (the value of the
`size` parameter)


On Tue, Aug 12, 2014 at 12:48 AM, Ashish Mishra 
wrote:

> I recently added a binary type field to all documents with mapping
> "store": "true".  The field contents are large and as a result the on-disk
> index size rose by 3x, from 2.5Gb/shard to ~8Gb/shard.
>
> After this change I've seen a big jump in query latency.  Searches which
> previously took 40-60ms now take 800ms and longer.  This is the case even
> for queries which *don't* return the binary field.
> I tried optimizing the index down to max_num_segments=1, but query latency
> remains high.
>
> Is this expected?  Obviously queries returning the new field will take a
> hit (since field data needs to be loaded from disk).  But I would've
> expected other queries should not be much affected.
>
> Is the problem that larger file sizes make memory-mapping and the FS cache
> less efficient?  Or are stored fields still getting loaded from disk even
> when not included in the "fields" term?
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/6ef50cab-3004-490b-bc2d-ea7e71a824a5%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Adrien Grand

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j572SMo4rHOBGLRNMwHEv35WskjqZGMgpiJedYxAOP6-g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Segfault in ffi_prep_closure_loc

Yeah its a Vmware VM but i have a couple of ES 1.1 sitting in another 
Cluster without Issues, are there known issues with running ES in a VM ?

Thanks Jörg! I'll update libffi the next couple of days

markus

Am Dienstag, 12. August 2014 15:12:16 UTC+2 schrieb Jörg Prante:
>
> Same configuration runs here, on bare metal. Maybe you run a VM?
>
> Just a shot in the dark, it looks like a bug in libffi, and there are more 
> recent versions like 3.0.5, so if you feel like hacking this bug, I would 
> try if libffi 3.1 from https://github.com/atgreen/libffi/ gives a 
> different picture.
>
> Jörg
>
>
> On Tue, Aug 12, 2014 at 2:59 PM, Markus Burger  > wrote:
>
>> exactly...
>>
>> [root@dev ~]# rpm -q libffi
>> libffi-3.0.5-3.2.el6.x86_64
>>
>> markus
>>
>> Am Dienstag, 12. August 2014 14:51:45 UTC+2 schrieb Jörg Prante:
>>>
>>> And 
>>>
>>> rpm -q libffi
>>>
>>> gives
>>>
>>> libffi-3.0.5-3.2.el6.x86_64
>>>
>>> ?
>>>
>>> Jörg
>>>
>>>
>>>
>>> On Tue, Aug 12, 2014 at 2:19 PM, Markus Burger  
>>> wrote:
>>>
 [root@dev limits.d]# runuser -s /bin/bash elasticsearch -c 'ulimit -l'
 unlimited
 [root@dev limits.d]# sysctl vm.max_map_count
 vm.max_map_count = 262144

 markus

 Am Dienstag, 12. August 2014 13:58:30 UTC+2 schrieb Jörg Prante:

> "S User" --> "ES user"
>
>  -- 
 You received this message because you are subscribed to the Google 
 Groups "elasticsearch" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to elasticsearc...@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/
 msgid/elasticsearch/044b2715-c566-476b-ba8f-035cb4cf0af6%
 40googlegroups.com 
 
 .

 For more options, visit https://groups.google.com/d/optout.

>>>
>>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/8485ac49-7d23-47f2-b28f-0b475052d293%40googlegroups.com
>>  
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b6e894da-6ebe-4f30-9a7b-7daf76b4b4d4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Can't delete Index!

The link doesn't work.
I am not sure about the difference, I have always deleted the same, curl 
XPUT http... //to create, and curl XDelete /http:...
Did the log tell you anything, there are some weird errors in there.

On Tuesday, August 12, 2014 3:04:39 AM UTC-4, Jörg Prante wrote:
>
> So you are wrestling with aliases. You can not delete aliases by file 
> system operations. Have you checked
>
>
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-aliases.html#deleting
>
> for deleting aliases?
>
> Jörg
>
>
> On Tue, Aug 12, 2014 at 4:10 AM, Sam2014 
> > wrote:
>
>> ///Attached is the cluster log that covers the whole sequence from 
>> shutting the cluster and re-activating it.
>>
>> //Show me current state of cluster
>> {
>>"cluster_name": "rexCluster",
>>"status": "red",
>>"timed_out": false,
>>"number_of_nodes": 2,
>>"number_of_data_nodes": 2,
>>"active_primary_shards": 7,
>>"active_shards": 7,
>>"relocating_shards": 0,
>>"initializing_shards": 2,
>>"unassigned_shards": 21
>> }
>>
>> //Show me indices on the cluster
>>
>> GET _aliases
>> {
>>"foo": {
>>   "aliases": {}
>>},
>>"mediregs": {
>>   "aliases": {}
>>},
>>"twitter": {
>>   "aliases": {}
>>}
>> }
>>
>>
>>
>> //Shut down the cluster
>>
>> POST /_cluster/nodes/_shutdown
>> {
>>"cluster_name": "rexCluster",
>>"nodes": {
>>   "_vPEfmc9TruIpugk4NWleg": {
>>  "name": "node 1"
>>   },
>>   "DB94rWiiSvS9HUTXwI7GTw": {
>>  "name": "node 2"
>>   }
>>}
>> }
>>
>>  //Go into the fs and delete files under /data
>>
>> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0$
>>  
>> ls
>>
>> *indices*  node.lock  *_state*
>>
>> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0$
>>  
>> cd indices/
>>
>> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$
>>  
>> ls
>>
>> *foo*  *mediregs*  *twitter*
>>
>> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$
>>  
>> sudo rm -rf foo
>>
>> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$
>>  
>> sudo rm -rf mediregs
>>
>> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$
>>  
>> sudo rm -rf twitter
>>
>> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$
>>  
>> ls
>>
>>
>> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$
>>  
>> //Restart nodes 1 &2 - should be in log file
>>
>> Get _aliases //Again - they came back!!
>> {
>>"foo": {
>>   "aliases": {}
>>},
>>"mediregs": {
>>   "aliases": {}
>>},
>>"twitter": {
>>   "aliases": {}
>>}
>> }
>>
>>
>> //Ok - Try to delete index
>> DELETE /mediregs
>>
>> //takes little while and comes back
>> {
>>"acknowledged": false
>> }
>>
>>
>>
>>
>> On Monday, August 11, 2014 7:12:22 PM UTC-4, Jörg Prante wrote:
>>
>>> Can you post your detailed steps and debug logs?
>>>
>>> Make also sure there is no third node - this looks like ES is shipping 
>>> shards to a third node from which the two nodes are recovering from.
>>>
>>> Jörg
>>>
>>>
>>> On Tue, Aug 12, 2014 at 1:06 AM, Sam2014  wrote:
>>>
 I stopped node 1, then I stopped node 2. I disabled the cluster, 
 deleted files from nodes, Made sure no Indices on either node. I 
 re-enabled 
 cluster in yml, restarted cluster, and the index came back!


 On Monday, August 11, 2014 6:16:42 PM UTC-4, Jörg Prante wrote:

> There is no vicious circle. First you must stop the cluster. Then, 
> after processes have exited, continue with file operations. After that, 
> you 
> can start the cluster nodes again.
>
> Jörg
>
>
> On Mon, Aug 11, 2014 at 11:49 PM, Sam2014  wrote:
>
>> Oops, When I restarted the cluster/nodes, I see shards under an 
>> "unassigned node", neither of the two nodes of the cluster. Is there a 
>> way 
>> I can route/assign /delete out of this vicious cycle?
>>
>>
>> On Monday, August 11, 2014 5:46:25 PM UTC-4, Sam2014 wrote:
>>>
>>> I also deleted the directory, restarted the nodes, and it recovered 
>>> the index still, which is null.
>>>
>>> On Monday, August 11, 2014 5:42:53 PM UTC-4, Sam2014 wrote:

 I wish it was that simple, I deleted the directory of the index 
 from /data. Yet, the index is still there and won't be deleted or 
 recreated.. I still get acknowledged = false.

 On Monday, August 11, 2014 5:31:36 PM UTC-4, Jörg Prante wrote:
>
> Your JVMs (at least one of them) are not able to process the 
> command, so you should check out the data directory where the file of 
> the 
> index to be deleted are located. Shut your cluster down, remove t

Re: Why my buckets's doc_count is limit to 3000?

I am not sure to understand if your question is about aggregations or
indexing speed?


On Tue, Aug 12, 2014 at 11:24 AM, 陈浩  wrote:

> this is my DSL
>
> {
>> "query":{
>> "match_all":{}
>> },
>>
>> "aggs": {
>> "range": {
>> "date_range": {
>> "field": "@timestamp", "format":
>> "yyy.MM.dd.HH.mm.ss",
>> "ranges": [{"from": "2014.08.12.09.18.45", "to":
>> "2014.08.12.09.20.50"}]
>> },
>> "aggs": {
>> "over_time": {
>> "date_histogram": {
>> "field": "@timestamp",
>> "interval": "1s",
>> "format": "yyy.MM.dd.HH.mm.ss"
>> },
>> "aggs": {
>> "total_sent": {
>> "sum": {"field": "bytes_sent"}
>> }
>> }
>> }
>> }
>> }
>> }
>> }
>
>
>
>
>
> thus is my result
>
> {
>
>- "took":6,
>- "timed_out":false,
>- "_shards":{
>   - "total":96,
>   - "successful":96,
>   - "failed":0
>},
>- "hits":{
>   - "total":258002,
>   - "max_score":0.0,
>   - "hits":[
>  ]
>},
>- "aggregations":{
>   - "range":{
>  - "buckets":[
> - {
>- "key":"2014.08.12.09.18.45-2014.08.12.09.20.50",
>- "from":1.407835125E12,
>- "from_as_string":"2014.08.12.09.18.45",
>- "to":1.40783525E12,
>- "to_as_string":"2014.08.12.09.20.50",
>- "doc_count":12000,
>- "over_time":{
>   - "buckets":[
>  - {
> - "key_as_string":"2014.08.12.09.18.45",
> - "key":1407835125000,
> - "doc_count":3000,
> - "total_sent":{
>- "value":6.6126308E7
> }
>  },
>  - {
> - "key_as_string":"2014.08.12.09.18.47",
> - "key":1407835127000,
> - "doc_count":3000,
> - "total_sent":{
>- "value":9.286586E7
> }
>  },
>  - {
> - "key_as_string":"2014.08.12.09.18.49",
> - "key":1407835129000,
> - "doc_count":3000,
> - "total_sent":{
>- "value":1.21316184E8
> }
>  },
>  - {
> - "key_as_string":"2014.08.12.09.18.51",
> - "key":1407835131000,
> - "doc_count":3000,
> - "total_sent":{
>- "value":8.3529544E7
> }
>  }
>   ]
>}
> }
>  ]
>   }
>}
>
> }
>
>
>
>
> my soft flow is:
> Nginx ==pipe==> syslogng ==udp==> logstash ==es_river==> elasticsearch
>
> How can I break the 3000doc/sec ?
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/1fa43dda-31cd-4f10-95bc-0f31778d9663%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Adrien Grand

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6kYQAU6ExwMFwxTuy3LQQx%2BLSh1_-wNO7cxTd9GbF-Ow%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Segfault in ffi_prep_closure_loc

Same configuration runs here, on bare metal. Maybe you run a VM?

Just a shot in the dark, it looks like a bug in libffi, and there are more
recent versions like 3.0.5, so if you feel like hacking this bug, I would
try if libffi 3.1 from https://github.com/atgreen/libffi/ gives a different
picture.

Jörg


On Tue, Aug 12, 2014 at 2:59 PM, Markus Burger 
wrote:

> exactly...
>
> [root@dev ~]# rpm -q libffi
> libffi-3.0.5-3.2.el6.x86_64
>
> markus
>
> Am Dienstag, 12. August 2014 14:51:45 UTC+2 schrieb Jörg Prante:
>>
>> And
>>
>> rpm -q libffi
>>
>> gives
>>
>> libffi-3.0.5-3.2.el6.x86_64
>>
>> ?
>>
>> Jörg
>>
>>
>>
>> On Tue, Aug 12, 2014 at 2:19 PM, Markus Burger 
>> wrote:
>>
>>> [root@dev limits.d]# runuser -s /bin/bash elasticsearch -c 'ulimit -l'
>>> unlimited
>>> [root@dev limits.d]# sysctl vm.max_map_count
>>> vm.max_map_count = 262144
>>>
>>> markus
>>>
>>> Am Dienstag, 12. August 2014 13:58:30 UTC+2 schrieb Jörg Prante:
>>>
 "S User" --> "ES user"

  --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/elasticsearch/044b2715-c566-476b-ba8f-035cb4cf0af6%
>>> 40googlegroups.com
>>> 
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/8485ac49-7d23-47f2-b28f-0b475052d293%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEHzP8kKyjO9yGgydpCes0Ko7wdey6gUe5GMsGELteHLg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: CSV River showing unexpected results in a 3 node cluster

I think you could set in elasticsearch.yml:

node.river: _none_

On nodes you don't want to allocate any river.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 12 août 2014 à 15:01:27, Sree (srssreej...@gmail.com) a écrit:

Hi all,

I have a 3 node cluster . One is Master and others are eligible for master when
the master node failed. I installed CSV River plugin in master node. The csv
files which need to process is also in master node. When i am running the CSV
river plugin from master , then it trying to execute the indexing in other
nodes and it fails in finding the csv files. Because csv files are in master.

Any pointers on this. Is this an expected behaviour ?

Thank you,

Srijith
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c64b88ce-1f3c-4966-bd42-d9845e91ef36%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/etPan.53ea11f8.2ca88611.18f0%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

CSV River showing unexpected results in a 3 node cluster

2014-08-12 Thread Sree

Hi all,

I have a 3 node cluster . One is Master and others are eligible for master 
when the master node failed. I installed CSV River plugin in master node. 
The csv files which need to process is also in master node. When i am 
running the CSV river plugin from master , then it trying to execute the 
indexing in other nodes and it fails in finding the csv files. Because csv 
files are in master.

Any pointers on this. Is this an expected behaviour ?

Thank you,

Srijith

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/c64b88ce-1f3c-4966-bd42-d9845e91ef36%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Segfault in ffi_prep_closure_loc

exactly...

[root@dev ~]# rpm -q libffi
libffi-3.0.5-3.2.el6.x86_64

markus

Am Dienstag, 12. August 2014 14:51:45 UTC+2 schrieb Jörg Prante:
>
> And 
>
> rpm -q libffi
>
> gives
>
> libffi-3.0.5-3.2.el6.x86_64
>
> ?
>
> Jörg
>
>
>
> On Tue, Aug 12, 2014 at 2:19 PM, Markus Burger  > wrote:
>
>> [root@dev limits.d]# runuser -s /bin/bash elasticsearch -c 'ulimit -l'
>> unlimited
>> [root@dev limits.d]# sysctl vm.max_map_count
>> vm.max_map_count = 262144
>>
>> markus
>>
>> Am Dienstag, 12. August 2014 13:58:30 UTC+2 schrieb Jörg Prante:
>>
>>> "S User" --> "ES user"
>>>
>>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/044b2715-c566-476b-ba8f-035cb4cf0af6%40googlegroups.com
>>  
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/8485ac49-7d23-47f2-b28f-0b475052d293%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Segfault in ffi_prep_closure_loc

And

rpm -q libffi

gives

libffi-3.0.5-3.2.el6.x86_64

?

Jörg



On Tue, Aug 12, 2014 at 2:19 PM, Markus Burger 
wrote:

> [root@dev limits.d]# runuser -s /bin/bash elasticsearch -c 'ulimit -l'
> unlimited
> [root@dev limits.d]# sysctl vm.max_map_count
> vm.max_map_count = 262144
>
> markus
>
> Am Dienstag, 12. August 2014 13:58:30 UTC+2 schrieb Jörg Prante:
>
>> "S User" --> "ES user"
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/044b2715-c566-476b-ba8f-035cb4cf0af6%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEWcVL8V%2BiAOn9am22h-B5hnQ%3D6aXswz_YwrqSeLgX-NA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: what's the difference between solr and elasticsearch in hdfs case?

And did you try it with elasticsearch? I mean indexing and still using the
service?

If you really hit an issue, you can think of allocating new index on dedicated
nodes and then move them to "live" nodes.
Using aliases would be even better so you'll be able to switch from one old
index to the new one without any downtime.

Another solution could be to index in a dedicated cluster and then use snapshot
and restore to backup from index cluster to search cluster.

But again, I'd make sure that I really have problems before trying to solve
problems I don't have.

My 2 cents.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 12 août 2014 à 13:43:13, Jianyi (phoenix.w.2...@gmail.com) a écrit:

Hi David,

Thanks for your reply.

I'm not talking about scaling out.

Our team has a project providing web services based on Lucene. The current
architecture indexes the documents to lucene through some kafka-like queues.
But we have a problem: when massive documents comes for indexing(eg:
creating index for some application for the first time with large scale
datas), the service becomes too slow. So we want to do some work about
offline-indexing, that is to prepare the index well offline and replace the
old index online when needed. We are looking for some solution based on
hdfs.

Do you have any idears about the offline-indexing above?

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/what-s-the-difference-between-solr-and-elasticsearch-in-hdfs-case-tp4061659p4061695.html

Sent from the ElasticSearch Users mailing list archive at Nabble.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/etPan.53ea085c.189a769b.18f0%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

[ANN] Elasticsearch File System River Plugin 1.3.0 released

Heya,


I'm pleased to announce the release of the Elasticsearch File System River 
Plugin, version 1.3.0.

FS River Plugin offers a simple way to index local files into elasticsearch..

https://github.com/dadoonet/fsriver/

Release Notes - fsriver - Version 1.3.0



Update:
 * [74] - Update to elasticsearch 1.3.0 
(https://github.com/dadoonet/fsriver/issues/74)




Issues, Pull requests, Feature requests are warmly welcome on fsriver project 
repository: https://github.com/dadoonet/fsriver/
For questions or comments around this plugin, feel free to use elasticsearch 
mailing list: https://groups.google.com/forum/#!forum/elasticsearch

Enjoy,

- David

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/53ea06c1.cf4cb40a.0776.1fc6SMTPIN_ADDED_MISSING%40gmr-mx.google.com.
For more options, visit https://groups.google.com/d/optout.

Re: Segfault in ffi_prep_closure_loc

[root@dev limits.d]# runuser -s /bin/bash elasticsearch -c 'ulimit -l'
unlimited
[root@dev limits.d]# sysctl vm.max_map_count
vm.max_map_count = 262144

markus

Am Dienstag, 12. August 2014 13:58:30 UTC+2 schrieb Jörg Prante:
>
> "S User" --> "ES user"
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/044b2715-c566-476b-ba8f-035cb4cf0af6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

How to receive part of the text field?

2014-08-12 Thread Dmitriy Bashkalin

Hello.
There is a field with large amount of text. How I can get part of value, 
such as 200KB? By analogy with the from () / size ().
Like as "Give me the first 200KB part of object with this Id"

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/bf664bcf-f936-4391-84d9-46cd26cefdd6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Segfault in ffi_prep_closure_loc

"S User" --> "ES user"

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE-up9o89szdPG6F%3DphZcLB3c2NpYchh%3DDqPeuHD5WxPA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Segfault in ffi_prep_closure_loc

OK. Can you check if your mem lock setting for the  S user allows to
reserve locked memory?

ulimit -l

This is by default on RHEL 64kb which should be changed to unlimited.

Jörg


On Tue, Aug 12, 2014 at 1:30 PM, Markus Burger 
wrote:

> sadly exactly the same error accours :/
>
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x7fae3d43440a, pid=21440, tid=140387243357952
> #
> # JRE version: Java(TM) SE Runtime Environment (7.0_60-b19) (build
> 1.7.0_60-b19)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.60-b09 mixed mode
> linux-amd64 compressed oops)
> # Problematic frame:
> # C  [jna8009017922749255641.tmp+0x1240a]  ffi_prep_closure_loc+0x1a
>
> Am Dienstag, 12. August 2014 12:12:00 UTC+2 schrieb Jörg Prante:
>>
>> Can you try with Java 7u60?
>>
>> Jörg
>>
>>
>> On Tue, Aug 12, 2014 at 11:58 AM, Markus Burger 
>> wrote:
>>
>>> hello guys,
>>>
>>> i tryed to install elasticsearch 1.3.1 today on a dev machine and got a
>>> problem related to mlockall that i have never seen before and cant wrap my
>>> head around.
>>>
>>> # A fatal error has been detected by the Java Runtime Environment:
>>> #
>>> #  SIGSEGV (0xb) at pc=0x7f09617e040a, pid=3143, tid=139678674036480
>>> #
>>> # JRE version: Java(TM) SE Runtime Environment (7.0_65-b17) (build
>>> 1.7.0_65-b17)
>>> # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode
>>> linux-amd64 compressed oops)
>>> # Problematic frame:
>>> # C  [jna1731590185807699933.tmp+0x1240a]  ffi_prep_closure_loc+0x1a
>>>
>>> this is on a vanilla install with just a few custom settings (mainly
>>> related to /tmp mounted noexec), sysctl and limit settings are ok (the full
>>> log is attached).
>>>
>>> has anybody seen this before or has any clue why this might be happening
>>> ?
>>>
>>> best regards, markus
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>>
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/elasticsearch/3ee4f06b-e207-4b4b-9b49-f908260d852e%
>>> 40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/5f339f21-df3e-406c-9947-68443df5d307%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEVze7Do%3DXFtso%3Dazm-1WVKNx%3D%3DkhPACEVmOsb%2BvMAp_Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: what's the difference between solr and elasticsearch in hdfs case?

Hi David,

Thanks for your reply.

I'm not talking about scaling out.

Our team has a project providing web services based on Lucene. The current
architecture indexes the documents to lucene through some kafka-like queues.
But we have a problem: when massive documents comes for indexing(eg:
creating index for some application for the first time with large scale
datas), the service becomes too slow. So we want to do some work about
offline-indexing, that is to prepare the index well offline and replace the
old index online when needed.  We are looking for some solution based on
hdfs.

Do you have any idears about the offline-indexing above?



--
View this message in context: 
http://elasticsearch-users.115913.n3.nabble.com/what-s-the-difference-between-solr-and-elasticsearch-in-hdfs-case-tp4061659p4061695.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1407842875709-4061695.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.

Re: what's the difference between solr and elasticsearch in hdfs case?

Hi David,

Thanks for your reply.

I'm not talking about scaling out.

Out team has a project providing web services based on Lucene. The current
architecture indexes the documents to lucene through some kafka-like queues.
But we have a problem: when massive documents comes for indexing(eg:
creating index for some application for the first time with large scale
datas), the service becomes too slow. So we want to do some work about
offline-indexing, that is to prepare the index well offline and replace the
old index online when needed.  We are looking for some solution based on
hdfs.

Do you have any idears about the offline-indexing above?



--
View this message in context: 
http://elasticsearch-users.115913.n3.nabble.com/what-s-the-difference-between-solr-and-elasticsearch-in-hdfs-case-tp4061659p4061684.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1407834529982-4061684.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.

Re: Segfault in ffi_prep_closure_loc

sadly exactly the same error accours :/

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x7fae3d43440a, pid=21440, tid=140387243357952
#
# JRE version: Java(TM) SE Runtime Environment (7.0_60-b19) (build 
1.7.0_60-b19)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.60-b09 mixed mode 
linux-amd64 compressed oops)
# Problematic frame:
# C  [jna8009017922749255641.tmp+0x1240a]  ffi_prep_closure_loc+0x1a

Am Dienstag, 12. August 2014 12:12:00 UTC+2 schrieb Jörg Prante:
>
> Can you try with Java 7u60?
>
> Jörg
>
>
> On Tue, Aug 12, 2014 at 11:58 AM, Markus Burger  > wrote:
>
>> hello guys,
>>
>> i tryed to install elasticsearch 1.3.1 today on a dev machine and got a 
>> problem related to mlockall that i have never seen before and cant wrap my 
>> head around.
>>
>> # A fatal error has been detected by the Java Runtime Environment:
>> #
>> #  SIGSEGV (0xb) at pc=0x7f09617e040a, pid=3143, tid=139678674036480
>> #
>> # JRE version: Java(TM) SE Runtime Environment (7.0_65-b17) (build 
>> 1.7.0_65-b17)
>> # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode 
>> linux-amd64 compressed oops)
>> # Problematic frame:
>> # C  [jna1731590185807699933.tmp+0x1240a]  ffi_prep_closure_loc+0x1a
>>
>> this is on a vanilla install with just a few custom settings (mainly 
>> related to /tmp mounted noexec), sysctl and limit settings are ok (the full 
>> log is attached).
>>
>> has anybody seen this before or has any clue why this might be happening ?
>>
>> best regards, markus
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/3ee4f06b-e207-4b4b-9b49-f908260d852e%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5f339f21-df3e-406c-9947-68443df5d307%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Segfault in ffi_prep_closure_loc

2014-08-12 Thread 'Sandeep Ramesh Khanzode' via elasticsearch

Can you try with Java 7u60?

Jörg


On Tue, Aug 12, 2014 at 11:58 AM, Markus Burger 
wrote:

> hello guys,
>
> i tryed to install elasticsearch 1.3.1 today on a dev machine and got a
> problem related to mlockall that i have never seen before and cant wrap my
> head around.
>
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x7f09617e040a, pid=3143, tid=139678674036480
> #
> # JRE version: Java(TM) SE Runtime Environment (7.0_65-b17) (build
> 1.7.0_65-b17)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode
> linux-amd64 compressed oops)
> # Problematic frame:
> # C  [jna1731590185807699933.tmp+0x1240a]  ffi_prep_closure_loc+0x1a
>
> this is on a vanilla install with just a few custom settings (mainly
> related to /tmp mounted noexec), sysctl and limit settings are ok (the full
> log is attached).
>
> has anybody seen this before or has any clue why this might be happening ?
>
> best regards, markus
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/3ee4f06b-e207-4b4b-9b49-f908260d852e%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG50%2B_CfH%2BnHpVLajRT63-4c1eCoi_MWgR5u7FHf871gg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

TermsLookupFilter Caching

Hi,

Does the TermsLookupFilter cache results in a bitmap/bitset? Or does it 
cache the results of the filter completely without using bits for document 
identifiers? 

Thanks,
Sandeep

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f3db7695-87d4-4e15-8174-dbecf5211b3b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Segfault in ffi_prep_closure_loc

hello guys,

i tryed to install elasticsearch 1.3.1 today on a dev machine and got a 
problem related to mlockall that i have never seen before and cant wrap my 
head around.

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x7f09617e040a, pid=3143, tid=139678674036480
#
# JRE version: Java(TM) SE Runtime Environment (7.0_65-b17) (build 
1.7.0_65-b17)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode 
linux-amd64 compressed oops)
# Problematic frame:
# C  [jna1731590185807699933.tmp+0x1240a]  ffi_prep_closure_loc+0x1a

this is on a vanilla install with just a few custom settings (mainly 
related to /tmp mounted noexec), sysctl and limit settings are ok (the full 
log is attached).

has anybody seen this before or has any clue why this might be happening ?

best regards, markus

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/3ee4f06b-e207-4b4b-9b49-f908260d852e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


segfault.elastic.hs_err
Description: Binary data

Re: Embedded ElasticSearch On Java

Replace 

.setTransportAddress(new InetSocketTransportAddress("localhost", 9300)); 

with

.addTransportAddress(new InetSocketTransportAddress('localhost', 9300)).

And I guess if you dont give cluster name, it automatically joins the 
default cluster.

I tried the code that you provided and changed above mentioned code. It 
works on my end. Can you try it?

On Monday, August 11, 2014 11:34:43 PM UTC+5:30, Kfeenz wrote:
>
> So I am very new to elasticsearch... so I apologize in advance..
>
> I started a local instance of elasticsearch and I am trying to connect to 
> it through the Java API.
>
> I was under the impression that the transport client was for remote 
> clients?
>
> I tried:
>
> @Test
> public void testIndexResponse() {
>
>   Client client = new TransportClient().setTransportAddress(new 
> InetSocketTransportAddress("localhost", 9300));
>
>   String json = "{" +
> "\"user\":\"kimchy\"," +
> "\"postDate\":\"2013-01-30\"," +
> "\"message\":\"trying out Elasticsearch\"" +
> "}";
>
>   IndexResponse response = client.prepareIndex("twitter", "tweet")
> .setSource(json)
> .execute()
> .actionGet();
>
>
>   client.close();
>
>   System.out.println(response.getIndex());
> }
>
> I receive org.elasticsearch.client.transport.NoNodeAvailableException: 
> None of the configured nodes are available: [].
>
>
>
> On Monday, August 11, 2014 1:19:06 PM UTC-4, Vivek Sachdeva wrote:
>>
>> Have you tried using transport client for connecting... 
>>
>> On Monday, August 11, 2014 10:26:29 PM UTC+5:30, Kfeenz wrote:
>>>
>>> All,
>>>
>>> I know this post is old, but I continue to have an issue with this... 
>>>
>>> I get an NoSuchMethodError: org.apache.log4j.Logger.isTraceEnabled()Z 
>>> exception when I run
>>>
>>> Node node = NodeBuilder.nodeBuilder().local(true).node(); // exception 
>>> thrown here...
>>> Client client = node.client();
>>>
>>> I have tried including several different slf4j and log4j libraries...
>>>
>>> Right now I have defined in my pom.xml:
>>>
>>> slf4j-api-1.7.7.jar
>>> slf4j-ext-1.7.7.jar
>>> log4j-core-2.0.jar
>>> log4j-api-2.0.jar
>>>
>>> Any suggestions?
>>> log4j-slf4j-impl-2.0.jar
>>>
>>> On Wednesday, November 30, 2011 6:55:57 PM UTC-5, Sam wrote:

 All

 I have a webapp (JAVA) with a list of items on the page. All these
 items are generated on html with session variables. I want to use
 ElasticSearch on the back end to do a search for items with different
 search criteria. Is there a sample I can use it as a resource?

 Thanks



-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/cc92ae76-4d7f-4ef3-8a29-1df4755ca3d0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ANN] Elasticsearch File System River Plugin 1.2.0 released

Heya,


I'm pleased to announce the release of the Elasticsearch File System River 
Plugin, version 1.2.0.

FS River Plugin offers a simple way to index local files into elasticsearch..

https://github.com/dadoonet/fsriver/

Release Notes - fsriver - Version 1.2.0


Fix:
 * [84] - Empty fs river definition throws an error 
(https://github.com/dadoonet/fsriver/issues/84)
 * [82] - File dates could be rounded to second or higher 
(https://github.com/dadoonet/fsriver/issues/82)
 * [75] - Adding new files while scanning could raise a non caught exception 
(https://github.com/dadoonet/fsriver/issues/75)
 * [60] - Files are not getting added to the index after copying files in the 
folder (https://github.com/dadoonet/fsriver/issues/60)
 * [32] - Files not correctly added/removed from index on Windows platform 
(https://github.com/dadoonet/fsriver/issues/32)

Update:
 * [83] - update_rate parameter uses TimeValue instead ms 
(https://github.com/dadoonet/fsriver/issues/83)
 * [81] - Tests: make tests more robust 
(https://github.com/dadoonet/fsriver/issues/81)
 * [79] - Clean tests and use hamcrest 
(https://github.com/dadoonet/fsriver/issues/79)
 * [78] - Update to JSch 0.1.51 (https://github.com/dadoonet/fsriver/issues/78)
 * [68] - Update to elasticsearch 1.2.0 
(https://github.com/dadoonet/fsriver/issues/68)

New:
 * [55] - support for ssh key instead of password 
(https://github.com/dadoonet/fsriver/issues/55)

Doc:
 * [80] - Docs: use SENSE like examples 
(https://github.com/dadoonet/fsriver/issues/80)
 * [77] - Docs: add debug information 
(https://github.com/dadoonet/fsriver/issues/77)


Issues, Pull requests, Feature requests are warmly welcome on fsriver project 
repository: https://github.com/dadoonet/fsriver/
For questions or comments around this plugin, feel free to use elasticsearch 
mailing list: https://groups.google.com/forum/#!forum/elasticsearch

Enjoy,

- David

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/53e9deb3.c46cb40a.4664.1882SMTPIN_ADDED_MISSING%40gmr-mx.google.com.
For more options, visit https://groups.google.com/d/optout.

Why my buckets's doc_count is limit to 3000?

2014-08-12 Thread 陈浩

this is my DSL

{
> "query":{
> "match_all":{}
> },
>
> "aggs": {
> "range": {
> "date_range": {
> "field": "@timestamp", "format": 
> "yyy.MM.dd.HH.mm.ss",
> "ranges": [{"from": "2014.08.12.09.18.45", "to": 
> "2014.08.12.09.20.50"}]
> },  
> "aggs": {
> "over_time": {
> "date_histogram": {
> "field": "@timestamp",
> "interval": "1s",
> "format": "yyy.MM.dd.HH.mm.ss"
> },
> "aggs": {
> "total_sent": {
> "sum": {"field": "bytes_sent"}
> }
> }
> }
> }
> }
> }
> }



 

thus is my result

{
   
   - "took":6,
   - "timed_out":false,
   - "_shards":{
  - "total":96,
  - "successful":96,
  - "failed":0
   },
   - "hits":{
  - "total":258002,
  - "max_score":0.0,
  - "hits":[
 ]
   },
   - "aggregations":{
  - "range":{
 - "buckets":[
- {
   - "key":"2014.08.12.09.18.45-2014.08.12.09.20.50",
   - "from":1.407835125E12,
   - "from_as_string":"2014.08.12.09.18.45",
   - "to":1.40783525E12,
   - "to_as_string":"2014.08.12.09.20.50",
   - "doc_count":12000,
   - "over_time":{
  - "buckets":[
 - {
- "key_as_string":"2014.08.12.09.18.45",
- "key":1407835125000,
- "doc_count":3000,
- "total_sent":{
   - "value":6.6126308E7
}
 },
 - {
- "key_as_string":"2014.08.12.09.18.47",
- "key":1407835127000,
- "doc_count":3000,
- "total_sent":{
   - "value":9.286586E7
}
 },
 - {
- "key_as_string":"2014.08.12.09.18.49",
- "key":1407835129000,
- "doc_count":3000,
- "total_sent":{
   - "value":1.21316184E8
}
 },
 - {
- "key_as_string":"2014.08.12.09.18.51",
- "key":1407835131000,
- "doc_count":3000,
- "total_sent":{
   - "value":8.3529544E7
}
 }
  ]
   }
}
 ]
  }
   }

}




my soft flow is:
Nginx ==pipe==> syslogng ==udp==> logstash ==es_river==> elasticsearch

How can I break the 3000doc/sec ?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1fa43dda-31cd-4f10-95bc-0f31778d9663%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: org.elasticsearch.search.aggregations docs

2014-08-12 Thread Isabel Drost-Fromm

On Tue, Aug 12, 2014 at 1:13 AM, Jeff Steinmetz  wrote:

> Although I was specifically talking about documentation for the Java
> search API.
>
> For example, there is this
>
> http://www.elasticsearch.org/guide/en/elasticsearch/client/java-api/current/java-facets.html
>
> But ... haven't found anything that covers the Aggregations replacement.
>

I see - I don't find it there as well. You might want to check the JavaDoc*
(org.elasticsearch.action.search.SearchRequestBuilder for adding
aggregations to search requests, org.elasticsearch.search.aggregations.*
for Aggregations specific docs).

When looking for examples, me personally, I like reading test cases (same
package as above, just in the maven test folder), look for tests that use
the Java client:

SearchResponse response = client().prepareSearch(".").addAggregation...

Isabel

* If the elasticsearch hosted Javadoc isn't accessible you can always build
it yourself from source with mvn javadoc:javadoc (located in the
target/site/apidocs directory after building).

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAFSgB-C7w-9PYXjyG6Eo_QN%3DJxA4kGDq%3DeQh%2BAN-4gAv6yy%2BZA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Alias creation getting slower

2014-08-12 Thread Ranjith Venkatesan

Hi, 
We are using Elasticsearch for our search application. currently we have 3 
master nodes and 5 data nodes in our cluster with 160 indices with 5 shards 
each and replica of 2. 

Currently we have 20k users. We will create alias for each user with alias name 
as user id. As the number of alias gets increased, time to add alias also 
getting increased consistently. Now aliascreation takes upto 10 secs for 
most of the time. 

Is there any workaround for this. 

PS : found similar case here 
.
 
But no solution for this. 

Thanks in advance 

Ranjith Venkatesan

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b18dd013-b857-480d-b6c8-e6123239bd23%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: query_string query that contains < and > symbols

2014-08-12 Thread Tihomir Lichev

That makes much more sense :)
Thanks, it works now!
And somewhere in the query_string docs should be mentioned that 
simple_query_string should be preferred in where applicable :-P

12 август 2014, вторник, 12:07:58 UTC+3, Jörg Prante написа:
>
> Once again, it is not related to "escaping". 
>
> Why don't you use simple query_string?
>
>
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-simple-query-string-query.html
>
> It wraps the crappy query_string into "match" query, correctly analyzed 
> and parsed to Elasticsearch settings, which query_string is not always able 
> to. And it provides boolean clauses.
>
> Jörg
>
> Examples:
>
> POST /test/_search
> {
> "query" : {
> "simple_query_string" : {
> "title" : {
> "query" : ">end"
> }
> }
> }
> }
>
> POST /test/_search
> {
> "query" : {
> "simple_query_string" : {
> "title" : {
> "query" : " testing and >end"
> }
> }
> }
> }
>
>
>
> On Tue, Aug 12, 2014 at 11:01 AM, Tihomir Lichev  > wrote:
>
>> I think I don't agree ...
>> I'm using query_string because I want to give the users the ability to 
>> use AND, OR +, - etc., out of the box.
>> I'm able to escape all other symbols except < and >, and I can use them 
>> as part of the field content, also as part of the query like any regular 
>> letter.
>> I dont understand why there is no way to escape those 2 symbols only. How 
>> they are different compared to the others ???
>>
>> Tihomir
>>
>> 12 август 2014, вторник, 11:57:21 UTC+3, Jörg Prante написа:
>>>
>>> This is not a problem of escape.
>>>
>>> Always use "match" query. Never use "query_string".
>>>
>>> Jörg
>>>
>>>
>>> On Tue, Aug 12, 2014 at 10:50 AM, Tihomir Lichev  
>>> wrote:
>>>
  Does anyone have an idea hot to escape < and > in a query ???

 07 август 2014, четвъртък, 18:27:08 UTC+3, Tihomir Lichev написа:

> Hello,
> I recently discovered very interesting problem
>
> The analyzer is *whitespace:*
> {
>   "mappings": {
> "test": {
>   "properties": {
> "title": {
>   "analyzer": "whitespace",
>   "type": "string"
> },
> "description": {
>   "analyzer": "whitespace",
>   "type": "string"
> }
>   }
> }
>   }
> }
>
>
> I have 3 documents with the following content:
> {
> "id": 1,
> "title" : "testing123",
> "description" : "a description"
> }
> {
> "id": 2,
> "title" : "testing >end",
> "description" : "another description"
> }
> {
> "id": 3,
> "title" : "testing  "description" : "another description"
> }
>
>
> Then I did the following search queries:
> {
>   "query": {
>
> "bool": {
>   "should": [
> {
>   "query_string": {
> "query": ">end",
> "fields": [
>   "title"
> ]
>   }
> }
>   ]
> }
>   }
> }
>
> RESULT: 3 hits (all docs) - wrong
> EXPLAIN description: "ConstantScore(title:{end TO *]), product of:"
>
> {
>   "query": {
>
> "bool": {
>   "should": [
> {
>   "query_string": {
> "query": " "fields": [
>   "title"
> ]
>   }
> }
>   ]
> }
>   }
> }
>
> RESULT: 3 hits (all docs) - wrong
> EXPLAIN description: "ConstantScore(title:[* TO end}), product of:"
>
> {
>   "query": {
>
> "bool": {
>   "should": [
> {
>   "query_string": {
> "query": "end",
> "fields": [
>   "title"
> ]
>   }
> }
>   ]
> }
>   }
> }
>
> RESULT: 0 hits (correct!)
>
> I tried to escape "<" and ">" in the query, but no success
>
> Search for < and > in the elasticsearch and lucene docs - also no 
> luck. Obvoiusly they are recognized as range operators, but I want to use 
> them also as normal text symbols.
> Is there any way to escape them properly ?
>
>
>  -- 
 You received this message because you are subscribed to the Google 
 Groups "elasticsearch" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to elasticsearc...@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/
 msgid/elasticsearch/9fee6a55-b3ff-4192-9329-4e5f6fead45e%
 40googlegroups.com

Re: Completion Suggester : Support for numeric value completion suggestion

2014-08-12 Thread Hemant Singh

Thanks Alexander,

It worked with Standard Analyzer. Will look for different analyzers, and 
chose the best which fits the use case. 

Thanks,
Hemant

On Tuesday, August 12, 2014 2:21:48 PM UTC+5:30, Alexander Reelsen wrote:
>
> Hey,
>
> The default analyzer used in the completion suggester is the simple one, 
> which strips out numbers.
>
>
> --Alex
>
>
> On Thu, Aug 7, 2014 at 1:43 PM, Hemant > 
> wrote:
>
>> Hello, 
>>
>> I was trying the following use cases using completion suggester - 
>> 1. Suggest Song on search song by id - numeric value(long)  - ***
>> 2. Suggest Song by Song Title and 
>> some more use cases, but that does not make any sense for this question. 
>>
>> But I was unable to get the first use case done. 
>>
>> I indexed the song as follows - 
>>
>> curl -X PUT 'localhost:9200/music/song/1?refresh=true' -d '{
>>
>> "name" : "Nevermind",
>>
>> "suggest" : {
>>
>> "input": [ "Nevermind", "Nirvana", "123" ],
>>
>> "output": "Nirvana - Nevermind",
>>
>> "payload" : { "artistId" : 2321 },
>>
>> "weight" : 34
>>
>> }
>> }'
>>
>> curl -X PUT 'localhost:9200/music/song/2?refresh=true' -d '{
>>
>> "name" : "Friday",
>>
>> "suggest" : {
>>
>> "input": [ "Friday", "Rebecca","1456" ],
>>
>> "output": "Nirvana - Nevermind",
>>
>> "payload" : { "artistId" : 2322 },
>>
>> "weight" : 34
>>
>> }
>> }'
>>
>> Now when I query - 
>>
>> curl -X POST 'localhost:9200/music/_suggest?pretty' -d '{
>>
>> "song-suggest" : {
>>
>> "text" : "123",
>>
>> "completion" : {
>>
>> "field" : "suggest"
>>
>> }
>> }
>> }'
>>
>> I expected the following result by the completion search result - 
>>
>> {
>>   "_shards" : {
>> "total" : 5,
>> "successful" : 5,
>> "failed" : 0
>>   },
>>   "song-suggest" : [ {
>> "text" : "123",
>> "offset" : 0,
>> "length" : 3,
>> "options" : [ {
>>   "text" : "Nirvana - Nevermind",
>>   "score" : 34.0, "payload" : {"artistId":2321}
>> } ]
>>   } ]
>> }
>>
>> But did not succeed. 
>>
>> From several attempt came to notice that, numeric value completion 
>> suggester is not yet supported in Elaticsearch. 
>>
>> After reading few forum posts, and Elasticsearch documentation, noticed 
>> that this completion suggester is only for terms which consists only 
>> alphabet, as it is using FST as it's internal data structure. 
>> Elasticsearch : you-complete-me blog 
>>  , 
>> Search-suggesters-completion 
>> article 
>> 
>>
>> Correct me, If I have understood something wrongly. 
>>
>> My question is, Will Elasticsearch be supporting numeric value in near 
>> future for completion suggester ?
>> Or is there any way I can use completion suggester for this purpose ?
>>
>> Or suggest me some better alternative to solve this problem. 
>>
>> Thanks,
>> Hemant
>>
>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/c3b74171-9a1e-4c32-8a4b-2d959390c61b%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/ab701dd1-0cc7-48dd-b0de-4c8ccc9ef7e3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: query_string query that contains < and > symbols

Once again, it is not related to "escaping".

Why don't you use simple query_string?

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-simple-query-string-query.html

It wraps the crappy query_string into "match" query, correctly analyzed and
parsed to Elasticsearch settings, which query_string is not always able to.
And it provides boolean clauses.

Jörg

Examples:

POST /test/_search
{
"query" : {
"simple_query_string" : {
"title" : {
"query" : ">end"
}
}
}
}

POST /test/_search
{
"query" : {
"simple_query_string" : {
"title" : {
"query" : " testing and >end"
}
}
}
}



On Tue, Aug 12, 2014 at 11:01 AM, Tihomir Lichev  wrote:

> I think I don't agree ...
> I'm using query_string because I want to give the users the ability to use
> AND, OR +, - etc., out of the box.
> I'm able to escape all other symbols except < and >, and I can use them as
> part of the field content, also as part of the query like any regular
> letter.
> I dont understand why there is no way to escape those 2 symbols only. How
> they are different compared to the others ???
>
> Tihomir
>
> 12 август 2014, вторник, 11:57:21 UTC+3, Jörg Prante написа:
>>
>> This is not a problem of escape.
>>
>> Always use "match" query. Never use "query_string".
>>
>> Jörg
>>
>>
>> On Tue, Aug 12, 2014 at 10:50 AM, Tihomir Lichev 
>> wrote:
>>
>>> Does anyone have an idea hot to escape < and > in a query ???
>>>
>>> 07 август 2014, четвъртък, 18:27:08 UTC+3, Tihomir Lichev написа:
>>>
 Hello,
 I recently discovered very interesting problem

 The analyzer is *whitespace:*
 {
   "mappings": {
 "test": {
   "properties": {
 "title": {
   "analyzer": "whitespace",
   "type": "string"
 },
 "description": {
   "analyzer": "whitespace",
   "type": "string"
 }
   }
 }
   }
 }


 I have 3 documents with the following content:
 {
 "id": 1,
 "title" : "testing123",
 "description" : "a description"
 }
 {
 "id": 2,
 "title" : "testing >end",
 "description" : "another description"
 }
 {
 "id": 3,
 "title" : "testing >>> "description" : "another description"
 }


 Then I did the following search queries:
 {
   "query": {

 "bool": {
   "should": [
 {
   "query_string": {
 "query": ">end",
 "fields": [
   "title"
 ]
   }
 }
   ]
 }
   }
 }

 RESULT: 3 hits (all docs) - wrong
 EXPLAIN description: "ConstantScore(title:{end TO *]), product of:"

 {
   "query": {

 "bool": {
   "should": [
 {
   "query_string": {
 "query": ">>> "fields": [
   "title"
 ]
   }
 }
   ]
 }
   }
 }

 RESULT: 3 hits (all docs) - wrong
 EXPLAIN description: "ConstantScore(title:[* TO end}), product of:"

 {
   "query": {

 "bool": {
   "should": [
 {
   "query_string": {
 "query": "end",
 "fields": [
   "title"
 ]
   }
 }
   ]
 }
   }
 }

 RESULT: 0 hits (correct!)

 I tried to escape "<" and ">" in the query, but no success

 Search for < and > in the elasticsearch and lucene docs - also no luck.
 Obvoiusly they are recognized as range operators, but I want to use them
 also as normal text symbols.
 Is there any way to escape them properly ?


  --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/elasticsearch/9fee6a55-b3ff-4192-9329-4e5f6fead45e%
>>> 40googlegroups.com
>>> 
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/d6bb86ed-111b-4909-a918-e8ae79757

Re: query_string query that contains < and > symbols

2014-08-12 Thread Tihomir Lichev

I think I don't agree ...
I'm using query_string because I want to give the users the ability to use 
AND, OR +, - etc., out of the box.
I'm able to escape all other symbols except < and >, and I can use them as 
part of the field content, also as part of the query like any regular 
letter.
I dont understand why there is no way to escape those 2 symbols only. How 
they are different compared to the others ???

Tihomir

12 август 2014, вторник, 11:57:21 UTC+3, Jörg Prante написа:
>
> This is not a problem of escape.
>
> Always use "match" query. Never use "query_string".
>
> Jörg
>
>
> On Tue, Aug 12, 2014 at 10:50 AM, Tihomir Lichev  > wrote:
>
>> Does anyone have an idea hot to escape < and > in a query ???
>>
>> 07 август 2014, четвъртък, 18:27:08 UTC+3, Tihomir Lichev написа:
>>
>>> Hello,
>>> I recently discovered very interesting problem
>>>
>>> The analyzer is *whitespace:*
>>> {
>>>   "mappings": {
>>> "test": {
>>>   "properties": {
>>> "title": {
>>>   "analyzer": "whitespace",
>>>   "type": "string"
>>> },
>>> "description": {
>>>   "analyzer": "whitespace",
>>>   "type": "string"
>>> }
>>>   }
>>> }
>>>   }
>>> }
>>>
>>>
>>> I have 3 documents with the following content:
>>> {
>>> "id": 1,
>>> "title" : "testing123",
>>> "description" : "a description"
>>> }
>>> {
>>> "id": 2,
>>> "title" : "testing >end",
>>> "description" : "another description"
>>> }
>>> {
>>> "id": 3,
>>> "title" : "testing >> "description" : "another description"
>>> }
>>>
>>>
>>> Then I did the following search queries:
>>> {
>>>   "query": {
>>>
>>> "bool": {
>>>   "should": [
>>> {
>>>   "query_string": {
>>> "query": ">end",
>>> "fields": [
>>>   "title"
>>> ]
>>>   }
>>> }
>>>   ]
>>> }
>>>   }
>>> }
>>>
>>> RESULT: 3 hits (all docs) - wrong
>>> EXPLAIN description: "ConstantScore(title:{end TO *]), product of:"
>>>
>>> {
>>>   "query": {
>>>
>>> "bool": {
>>>   "should": [
>>> {
>>>   "query_string": {
>>> "query": ">> "fields": [
>>>   "title"
>>> ]
>>>   }
>>> }
>>>   ]
>>> }
>>>   }
>>> }
>>>
>>> RESULT: 3 hits (all docs) - wrong
>>> EXPLAIN description: "ConstantScore(title:[* TO end}), product of:"
>>>
>>> {
>>>   "query": {
>>>
>>> "bool": {
>>>   "should": [
>>> {
>>>   "query_string": {
>>> "query": "end",
>>> "fields": [
>>>   "title"
>>> ]
>>>   }
>>> }
>>>   ]
>>> }
>>>   }
>>> }
>>>
>>> RESULT: 0 hits (correct!)
>>>
>>> I tried to escape "<" and ">" in the query, but no success
>>>
>>> Search for < and > in the elasticsearch and lucene docs - also no luck. 
>>> Obvoiusly they are recognized as range operators, but I want to use them 
>>> also as normal text symbols.
>>> Is there any way to escape them properly ?
>>>
>>>
>>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/9fee6a55-b3ff-4192-9329-4e5f6fead45e%40googlegroups.com
>>  
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d6bb86ed-111b-4909-a918-e8ae797579da%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: query_string query that contains < and > symbols

This is not a problem of escape.

Always use "match" query. Never use "query_string".

Jörg


On Tue, Aug 12, 2014 at 10:50 AM, Tihomir Lichev  wrote:

> Does anyone have an idea hot to escape < and > in a query ???
>
> 07 август 2014, четвъртък, 18:27:08 UTC+3, Tihomir Lichev написа:
>
>> Hello,
>> I recently discovered very interesting problem
>>
>> The analyzer is *whitespace:*
>> {
>>   "mappings": {
>> "test": {
>>   "properties": {
>> "title": {
>>   "analyzer": "whitespace",
>>   "type": "string"
>> },
>> "description": {
>>   "analyzer": "whitespace",
>>   "type": "string"
>> }
>>   }
>> }
>>   }
>> }
>>
>>
>> I have 3 documents with the following content:
>> {
>> "id": 1,
>> "title" : "testing123",
>> "description" : "a description"
>> }
>> {
>> "id": 2,
>> "title" : "testing >end",
>> "description" : "another description"
>> }
>> {
>> "id": 3,
>> "title" : "testing > "description" : "another description"
>> }
>>
>>
>> Then I did the following search queries:
>> {
>>   "query": {
>>
>> "bool": {
>>   "should": [
>> {
>>   "query_string": {
>> "query": ">end",
>> "fields": [
>>   "title"
>> ]
>>   }
>> }
>>   ]
>> }
>>   }
>> }
>>
>> RESULT: 3 hits (all docs) - wrong
>> EXPLAIN description: "ConstantScore(title:{end TO *]), product of:"
>>
>> {
>>   "query": {
>>
>> "bool": {
>>   "should": [
>> {
>>   "query_string": {
>> "query": "> "fields": [
>>   "title"
>> ]
>>   }
>> }
>>   ]
>> }
>>   }
>> }
>>
>> RESULT: 3 hits (all docs) - wrong
>> EXPLAIN description: "ConstantScore(title:[* TO end}), product of:"
>>
>> {
>>   "query": {
>>
>> "bool": {
>>   "should": [
>> {
>>   "query_string": {
>> "query": "end",
>> "fields": [
>>   "title"
>> ]
>>   }
>> }
>>   ]
>> }
>>   }
>> }
>>
>> RESULT: 0 hits (correct!)
>>
>> I tried to escape "<" and ">" in the query, but no success
>>
>> Search for < and > in the elasticsearch and lucene docs - also no luck.
>> Obvoiusly they are recognized as range operators, but I want to use them
>> also as normal text symbols.
>> Is there any way to escape them properly ?
>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/9fee6a55-b3ff-4192-9329-4e5f6fead45e%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEfEhK%3D-XWyy8kqRqN6_ESNGa7U1uuowDmm37_0%2BkXo0Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Completion Suggester : Support for numeric value completion suggestion

2014-08-12 Thread Alexander Reelsen

Hey,

The default analyzer used in the completion suggester is the simple one,
which strips out numbers.


--Alex


On Thu, Aug 7, 2014 at 1:43 PM, Hemant  wrote:

> Hello,
>
> I was trying the following use cases using completion suggester -
> 1. Suggest Song on search song by id - numeric value(long)  - ***
> 2. Suggest Song by Song Title and
> some more use cases, but that does not make any sense for this question.
>
> But I was unable to get the first use case done.
>
> I indexed the song as follows -
>
> curl -X PUT 'localhost:9200/music/song/1?refresh=true' -d '{
> "name" : "Nevermind",
> "suggest" : {
> "input": [ "Nevermind", "Nirvana", "123" ],
> "output": "Nirvana - Nevermind",
> "payload" : { "artistId" : 2321 },
> "weight" : 34
> }
> }'
>
> curl -X PUT 'localhost:9200/music/song/2?refresh=true' -d '{
> "name" : "Friday",
> "suggest" : {
> "input": [ "Friday", "Rebecca","1456" ],
> "output": "Nirvana - Nevermind",
> "payload" : { "artistId" : 2322 },
> "weight" : 34
> }
> }'
>
> Now when I query -
>
> curl -X POST 'localhost:9200/music/_suggest?pretty' -d '{
> "song-suggest" : {
> "text" : "123",
> "completion" : {
> "field" : "suggest"
> }
> }
> }'
>
> I expected the following result by the completion search result -
>
> {
>   "_shards" : {
> "total" : 5,
> "successful" : 5,
> "failed" : 0
>   },
>   "song-suggest" : [ {
> "text" : "123",
> "offset" : 0,
> "length" : 3,
> "options" : [ {
>   "text" : "Nirvana - Nevermind",
>   "score" : 34.0, "payload" : {"artistId":2321}
> } ]
>   } ]
> }
>
> But did not succeed.
>
> From several attempt came to notice that, numeric value completion
> suggester is not yet supported in Elaticsearch.
>
> After reading few forum posts, and Elasticsearch documentation, noticed
> that this completion suggester is only for terms which consists only
> alphabet, as it is using FST as it's internal data structure.
> Elasticsearch : you-complete-me blog
>  , 
> Search-suggesters-completion
> article
> 
>
> Correct me, If I have understood something wrongly.
>
> My question is, Will Elasticsearch be supporting numeric value in near
> future for completion suggester ?
> Or is there any way I can use completion suggester for this purpose ?
>
> Or suggest me some better alternative to solve this problem.
>
> Thanks,
> Hemant
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/c3b74171-9a1e-4c32-8a4b-2d959390c61b%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGCwEM9izOuN02APiS7vtha_fjbwNJd9%2Be4isquYaBzmNT9HWQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: query_string query that contains < and > symbols

2014-08-12 Thread Tihomir Lichev

Does anyone have an idea hot to escape < and > in a query ???

07 август 2014, четвъртък, 18:27:08 UTC+3, Tihomir Lichev написа:
>
> Hello,
> I recently discovered very interesting problem
>
> The analyzer is *whitespace:*
> {
>   "mappings": {
> "test": {
>   "properties": {
> "title": {
>   "analyzer": "whitespace",
>   "type": "string"
> },
> "description": {
>   "analyzer": "whitespace",
>   "type": "string"
> }
>   }
> }
>   }
> }
>
>
> I have 3 documents with the following content:
> {
> "id": 1,
> "title" : "testing123",
> "description" : "a description"
> }
> {
> "id": 2,
> "title" : "testing >end",
> "description" : "another description"
> }
> {
> "id": 3,
> "title" : "testing  "description" : "another description"
> }
>
>
> Then I did the following search queries:
> {
>   "query": {
>
> "bool": {
>   "should": [
> {
>   "query_string": {
> "query": ">end",
> "fields": [
>   "title"
> ]
>   }
> }
>   ]
> }
>   }
> }
>
> RESULT: 3 hits (all docs) - wrong
> EXPLAIN description: "ConstantScore(title:{end TO *]), product of:"
>
> {
>   "query": {
>
> "bool": {
>   "should": [
> {
>   "query_string": {
> "query": " "fields": [
>   "title"
> ]
>   }
> }
>   ]
> }
>   }
> }
>
> RESULT: 3 hits (all docs) - wrong
> EXPLAIN description: "ConstantScore(title:[* TO end}), product of:"
>
> {
>   "query": {
>
> "bool": {
>   "should": [
> {
>   "query_string": {
> "query": "end",
> "fields": [
>   "title"
> ]
>   }
> }
>   ]
> }
>   }
> }
>
> RESULT: 0 hits (correct!)
>
> I tried to escape "<" and ">" in the query, but no success
>
> Search for < and > in the elasticsearch and lucene docs - also no luck. 
> Obvoiusly they are recognized as range operators, but I want to use them 
> also as normal text symbols.
> Is there any way to escape them properly ?
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/9fee6a55-b3ff-4192-9329-4e5f6fead45e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: what's the difference between solr and elasticsearch in hdfs case?

I would not store my indices on HDFS. :)
Too slow IMHO.

Use local storage and let elasticsearch distribute your data over multiple
machines. Basically, with elasticsearch you don't need HDFS to scale out.
I don't know for SOLR.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 12 août 2014 à 10:22:03, Jianyi (phoenix.w.2...@gmail.com) a écrit:

Hi~

I'm new to both solr and elasticsearch. I have read that both the two
support creating index on hdfs.

So, what's the difference between solr and elasticsearch in hdfs case?

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/what-s-the-difference-between-solr-and-elasticsearch-in-hdfs-case-tp4061659.html

Sent from the ElasticSearch Users mailing list archive at Nabble.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/etPan.53e9d06b.79838cb2.18f0%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

what's the difference between solr and elasticsearch in hdfs case?

2014-08-12 Thread 'Sandeep Ramesh Khanzode' via elasticsearch

Hi~

I'm new to both solr and elasticsearch. I have read that both the two
support creating index on hdfs.

So, what's the difference between solr and elasticsearch in hdfs case?



--
View this message in context: 
http://elasticsearch-users.115913.n3.nabble.com/what-s-the-difference-between-solr-and-elasticsearch-in-hdfs-case-tp4061659.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1407811639795-4061659.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.

http://www.elasticsearch.org/ can't be visit

2014-08-12 Thread 王星龙

http://www.elasticsearch.org/ can't be visit

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/3524dbbd-21d7-4381-9071-467f30f7d3e9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: clarity for shard allocation disable/enable during upgrade

2014-08-12 Thread Clinton Gormley



On Monday, 11 August 2014 15:31:28 UTC+2, bitsof...@gmail.com wrote:
>
> I have 8 data nodes and 6 coordinator nodes in an active cluster running 
> 1.2.1
>
> I want to upgrade to 1.3.1
>
> When reading 
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html
>  
> the upgrade docs am I correct to assume:
>
> a) disable shard allocation before doing anything
>
> b) proceed to upgrade each node to 1.3.1
>
> c) only after ALL nodes are @ 1.3.1 then I can re-enable shard allocation.
>
> My question is that at some point during the upgrade of all the data 
> nodes, the shards on them will be "unassigned" and the cluster will not 
> function... correct?
>
> So in other words running some nodes as 1.2.1 and others as 1.3.1 with 
> shard allocation *enabled* is NOT advised and in general cluster 
> un-availability is expected due to shards being in an unassigned state as 
> each data node is upgraded.
>
> At least this is the behavior I see today, (not during an upgrade) when I 
> disable allocation and restart a node, those shards are unassigned until I 
> re-enable allocation
>
>
No, the procedure outlined above is not correct and would indeed result in 
unassigned shards, as you suspect.  Instead, you should:

1. Disable allocation
2. Upgrade ONE node
3. Reenable allocation
4. Wait for green
5. Repeat

Even when following the above process, you will likely end up with shards 
being copied over from one node to another (once allocation has been 
reenabled).  After restart, a replica will only reuse the segments that are 
exactly the same as those in the primary.  However, because primaries and 
replicas refresh, flush, and merge at different times, shards diverge from 
each other over time. The longer it has been since a replica was copied 
over from the primary, the fewer identical segments they will have in 
common.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/11f7f772-9d86-4a84-a523-a047f4bb88b4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

BitSet Filters in ES/Lucene



Hi,
I have looked at TermsLookupFilter and it is a good approach to cache 
frequently used filters. However, even if I write a custom filter plugin, I 
cannot use a BitSet to hold any sort of document identifier. Even the _uid 
field is converted into a TermFilter.

Assume a scenario where I need to tag millions of documents with a tag like 
"Finance", "IT", "Legal", etc.

Unless, I can cache these filters in memory, the cost of constructing this 
filter at run time per query is not practical. If I could map the documents 
to a numeric long identifier and put them in a BitMap, I could then cache 
them because the size reduces drastically. However, I cannot use this 
numeric long identifier in ES/Lucene filters, either Custom Filter Plugin 
or Terms Lookup Filter. Is there any way?

I read about possible solutions in ES and found this link: 
http://www.elasticsearch.org/blog/all-about-elasticsearch-filter-bitsets/. 

Please help with this scenario. Thanks,

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d2a12986-220b-44c8-ac8f-a836de692c16%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Just a bit of community fun! Post your node names!

2014-08-12 Thread Marion Faugoin


>
> Thor Loki Odin etc... 
>

:p 

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/3c47fa3f-6020-4c36-8d09-3170a526acce%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Elasticsearch - using the path hierarchy tokenizer to access different level of categories

2014-08-12 Thread Ramy



I'm very new in Elasticsearch and have a question about the hierarchical 
tokenizer of a path. Here is my code example:

My mapping code:

PUT /my_index
{
  "settings": {
"analysis": {
  "analyzer": {
"path-analyzer": {
  "type": "custom",
  "tokenizer": "path-tokenizer"
}
  },
  "tokenizer": {
"path-tokenizer": {
  "type": "path_hierarchy",
  "delimiter": "."
}
  }
}
  },
  "mappings": {
"my_type": {
  "dynamic": "strict",
  "properties": {
"group_path": {
  "type": "string",
  "index_analyzer": "path-analyzer",
  "search_analyzer": "keyword"
}
  }
}
  }
}


This is my PUT:

PUT /my_index/my_type/1
{
  "group_path": ["Book.Thriller.Adult","DVD.Comedy.Kids"]
}


This is my Query:

GET /my_index/my_type/_search?search_type=count
{
   "aggs": {
  "category": {
 "terms": {
"field": "group_path",
"size": 0
 }
  }
   }
}


And the result:

{
   ...
   "aggregations": {
  "category": {
 "buckets": [
{
   "key": "Book",
   "doc_count": 1
},
{
   "key": "Book.Thriller",
   "doc_count": 1
},
{
   "key": "Book.Thriller.Adult",
   "doc_count": 1
},
{
   "key": "DVD",
   "doc_count": 1
},
{
   "key": "DVD.Comedy",
   "doc_count": 1
},
{
   "key": "DVD.Comedy.Kids",
   "doc_count": 1
}
 ]
  }
   }
}


So far is everything good. What I'm looking for is that how can I create 
buckets? for example only for the first category. How can I get result like 
that?

{
   ...
   "aggregations": {
  "category": {
 "buckets": [
{
   "key": "Book",
   "doc_count": 1
},
{
   "key": "DVD",
   "doc_count": 1
}
 ]
  }
   }
}


Thank you for any help.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/454cb83e-c70a-4c31-b85e-06258b117a4d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Can't delete Index!

So you are wrestling with aliases. You can not delete aliases by file
system operations. Have you checked

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-aliases.html#deleting

for deleting aliases?

Jörg


On Tue, Aug 12, 2014 at 4:10 AM, Sam2014  wrote:

> ///Attached is the cluster log that covers the whole sequence from
> shutting the cluster and re-activating it.
>
> //Show me current state of cluster
> {
>   "cluster_name": "rexCluster",
>   "status": "red",
>   "timed_out": false,
>   "number_of_nodes": 2,
>   "number_of_data_nodes": 2,
>   "active_primary_shards": 7,
>   "active_shards": 7,
>   "relocating_shards": 0,
>   "initializing_shards": 2,
>   "unassigned_shards": 21
> }
>
> //Show me indices on the cluster
>
> GET _aliases
> {
>"foo": {
>   "aliases": {}
>},
>"mediregs": {
>   "aliases": {}
>},
>"twitter": {
>   "aliases": {}
>}
> }
>
>
>
> //Shut down the cluster
>
> POST /_cluster/nodes/_shutdown
> {
>"cluster_name": "rexCluster",
>"nodes": {
>   "_vPEfmc9TruIpugk4NWleg": {
>  "name": "node 1"
>   },
>   "DB94rWiiSvS9HUTXwI7GTw": {
>  "name": "node 2"
>   }
>}
> }
>
> //Go into the fs and delete files under /data
>
> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0$
> ls
>
> *indices*  node.lock  *_state*
>
> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0$
> cd indices/
>
> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$
> ls
>
> *foo*  *mediregs*  *twitter*
>
> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$
> sudo rm -rf foo
>
> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$
> sudo rm -rf mediregs
>
> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$
> sudo rm -rf twitter
>
> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$
> ls
>
> ubuntu@domU-12-31-39-0A-99-EA
> :/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$
> //Restart nodes 1 &2 - should be in log file
>
> Get _aliases //Again - they came back!!
> {
>"foo": {
>   "aliases": {}
>},
>"mediregs": {
>   "aliases": {}
>},
>"twitter": {
>   "aliases": {}
>}
> }
>
>
> //Ok - Try to delete index
> DELETE /mediregs
>
> //takes little while and comes back
> {
>"acknowledged": false
> }
>
>
>
>
> On Monday, August 11, 2014 7:12:22 PM UTC-4, Jörg Prante wrote:
>
>> Can you post your detailed steps and debug logs?
>>
>> Make also sure there is no third node - this looks like ES is shipping
>> shards to a third node from which the two nodes are recovering from.
>>
>> Jörg
>>
>>
>> On Tue, Aug 12, 2014 at 1:06 AM, Sam2014  wrote:
>>
>>> I stopped node 1, then I stopped node 2. I disabled the cluster, deleted
>>> files from nodes, Made sure no Indices on either node. I re-enabled cluster
>>> in yml, restarted cluster, and the index came back!
>>>
>>>
>>> On Monday, August 11, 2014 6:16:42 PM UTC-4, Jörg Prante wrote:
>>>
 There is no vicious circle. First you must stop the cluster. Then,
 after processes have exited, continue with file operations. After that, you
 can start the cluster nodes again.

 Jörg


 On Mon, Aug 11, 2014 at 11:49 PM, Sam2014  wrote:

> Oops, When I restarted the cluster/nodes, I see shards under an
> "unassigned node", neither of the two nodes of the cluster. Is there a way
> I can route/assign /delete out of this vicious cycle?
>
>
> On Monday, August 11, 2014 5:46:25 PM UTC-4, Sam2014 wrote:
>>
>> I also deleted the directory, restarted the nodes, and it recovered
>> the index still, which is null.
>>
>> On Monday, August 11, 2014 5:42:53 PM UTC-4, Sam2014 wrote:
>>>
>>> I wish it was that simple, I deleted the directory of the index from
>>> /data. Yet, the index is still there and won't be deleted or 
>>> recreated.. I
>>> still get acknowledged = false.
>>>
>>> On Monday, August 11, 2014 5:31:36 PM UTC-4, Jörg Prante wrote:

 Your JVMs (at least one of them) are not able to process the
 command, so you should check out the data directory where the file of 
 the
 index to be deleted are located. Shut your cluster down, remove the 
 index
 files of the index to be deleted from the filesystem, and start the 
 cluster
 up again.

 To avoid such situations, find out why the JVMs could not react,
 most probably there are exceptions in the server logs, and fix the 
 trouble.

 Jörg


 On Mon, Aug 11, 2014 at 11:24 PM, Sam2014 
 wrote:

> I got myself in trouble by changing the mapping.
>
> I am on a cluster of 2 nodes, and

Re: Role of dedicated master nodes when not elected