Re: Corrupt Cluster State

Thanks for the reply Boaz. Sorry for the late reply, was away for a while. We run ES ver 1.2.2 and use the zulu openJDK from azul. So I'm going to get you the mappings in another post, but I think the cluster ended up becoming corrupt from an incident we saw about 1 1/2 weeks ago. I started ano

Re: [Hadoop][pig] How to set the document id in ESStorage?

Yep.Working good for me :) Thanks. On Tuesday, September 2, 2014 4:47:41 PM UTC+5:30, Costin Leau wrote: > > You can through the 'es.mapping.id' parameter; see [1] > > Cheers, > > [1] > http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/configuration.html#_mapping > > > On 9/2

Server Side Scripting capability support in future?

Is there plan to add server side scripting like SQL StoredProc ? Example if we need to do further aggregations or store or return values and do some more mathematical calucaltions using variables. It would be great if it is allowed on server side rather than doing it on the client side. The perc

Re: Corrupted Shard on Recovery

Have you checked your hardware status as the error mentioned? I'd also do a FS check to be safe. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 3 September 2014 14:58, David Kleiner wrote: > Greetings, > > I tried

Re: excluding fields with partial fields not working?

got it solved? On Thursday, August 15, 2013 4:50:40 AM UTC+3, concept47 wrote: > > I'm trying to use the fields setting > to > exclude a particular field from search queries, specifically using partial > fields. > All my fields a

Corrupted Shard on Recovery

Greetings, I tried to overcome slowly recovering replica set, changed the number of replicas on index to 0, then to 1, getting this exception: -

Re: Elasticsearch and Hive work together

Could you give a specific use case that combining Hive and ES to enhance each other ?. I wonder that why don't we just use ES to retrieve data, big data ? Thank you. On Mon, Sep 1, 2014 at 11:02 PM, Costin Leau wrote: > Actually they are completely different. > Hive is a library built on top of

Re: aggregation of hierchical elements possible?

Hello Markus , Can you also paste what is returned too. Also this is what i had in mind. - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-pathhierarchy-tokenizer.html#analysis-pathhierarchy-tokenizer Thanks Vineeth On Tue, Sep 2, 2014 at 11:36 PM, Mark

Warn/Error message of Elasticsearch perl client

Hi Guys, I have been using the Elasticsearch's perl client for query, it gives me the following warning/Error messages: while I was running a perl program which uses ES perl client. ES still

How does the micro analysis of the Kibana works?

Hi, my question is how does the micro analysis of the Kibana works? I mean how it counts the values of the fields as exact value without setting 'not_analyzed' to the fields? Thank you. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubs

Re: Looking for Elasticsearch projects

We could always use help with CirrisSearch. It is the open source project that links MediaWiki to Elasticsearch. We have it installed on all the wikis at the wikimedia foundation but it isn't the default search backend on the largest ones yet. "Selling" points: Huge user community Basic queries wo

Looking for Elasticsearch projects

For those that are not regulars on the mailing list, I am a fairly active member that has used Elasticsearch for years. I am leaving my full-time job to focus on other (techie and non-techie) goals and would love to work on some interesting projects part-time. It can be either paid assignments or

[Hadoop] Running elasticsearch-hive in AWS EMR

x27;es_host:7104'); OK Time taken: 0.423 seconds hive> insert overwrite table table_es > select other_properties['account_owner_id'], other_properties['customer_id'], other_properties['update_operation'], other_properties['update_type'], other_propertie

moustache search templates are too limited

I want to use search templates to generate a query based on the value of a parameter (not its presence or being True). For example I have a sortType parameter that can be {nearest, cheapest, easiest, newest}. Each of these options causes a drastically different change to the query sent to ES.

Re: Transport Client connectedNodes() duplicates

H that's weird, never was aware of that... I always thought it was a hybrid tcp46 port issue. Will check if it is the same on my linux machines. Jörg On Tue, Sep 2, 2014 at 10:23 PM, Stefan Will wrote: > But I’m using unicast… ("discovery.zen.ping.multicast: false”), on both > the server,

Re: writing a custom scoring plugin

Not sure if you looked at the source code of the function score weighted boost plugin. The plugin uses query-time parameterization but it is of course very easy to modify this and use settings at ES node startup time for function score boosting, or a refresh mechanism that fetches parameters from w

Re: Need help/suggestion for the massive queries user case

Yes, This works fine. The query request will eventually get processed. It just throws this message. Thanks. On Tuesday, September 2, 2014 4:51:08 PM UTC-4, David Pilato wrote: > > I have no idea. I don't know the Perl Client. > > Does this works? > > curl localhost:9200 > > > > -- > *David Pi

Re: Need help/suggestion for the massive queries user case

I have no idea. I don't know the Perl Client. Does this works? curl localhost:9200  --  David Pilato | Technical Advocate | Elasticsearch.com @dadoonet | @elasticsearchfr Le 2 septembre 2014 à 22:46:15, Yuheng Du (yuheng.du.h...@gmail.com) a écrit: Tried to increase the memory for the VM, n

Re: Need help/suggestion for the massive queries user case

Tried to increase the memory for the VM, now ES runs ok, but still gives me error: Which is caused by the Elasticsearch's perl module's Try::Tiny module. How to remove this kind of warning/err

Re: Transport Client connectedNodes() duplicates

But I’m using unicast… ("discovery.zen.ping.multicast: false”), on both the server, and the client. Also, after setting "network.host: _non_loopback:ipv4_”, the ES node is no longer binding to the wildcard (0), but directly to my local LAN address. I can confirm using netstat: tcp4       0 

Re: Spring Elasticsearch vs Spring Data Elasticsearch:

Thanks David, I'll play with Spring Elasticsearch project first. I have used other Spring Data projects, it seems that I don't need all these feature for now, Jinyuan (Jack) Zhou On Tue, Sep 2, 2014 at 12:56 PM, David Pilato wrote: > As the second project author, I can say what it provides: >

Re: Sense on github abandoned?

Hi Demian, Thx for the suggestion. The tricky part is that it will be search specific, so we'd need to extend the action menu to be request sensitive. It also means we'd need to walk all request and decides what makes sense there. I think it's has merit but will require work to implement. Cheers,

Re: Spring Elasticsearch vs Spring Data Elasticsearch:

As the second project author, I can say what it provides: Automatic index/mapping/templates/aliases création when starting (as Hibernate does somehow). Access to a client (Transport/Node) - by annotation like @Inject Client client. Access to a Node You can look at tests packages to see how it ac

Spring Elasticsearch vs Spring Data Elasticsearch:

Hi, I'm planning to use java native client to mainly to bulk index a lot of data. I see there are two project listed on the elasticsearch guide's integration page. one is a sub project of spring data at https://github.com/spring-projects/spring-data-elasticsearch. One is a third party project

Re: Transport Client connectedNodes() duplicates

When I start a default ES node on Mac OSX, it opens on address "0" ("bind to all interfaces") a server socket, so I can see this in netstat tcp46 0 0 *.9300 *.*LISTEN This means a TCP6 port (in hybrid mode with TCP4 stack compatbility) is used. For t

Re: Need help/suggestion for the massive queries user case

You don't have enough memory available on your laptop I guess... -- David ;-) Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 2 sept. 2014 à 21:05, Yuheng Du a écrit : Ok. you are right. I was able to set the maximum heap size and now it looks like: "jvm" : { "timestamp" : 140968439

Re: Add field to all documents of an index.

Thanks! That's what I have been thinking! :) On 2 September 2014 13:15, vineeth mohan wrote: > Hello John , > > Update by query plugin would be a good choice - > https://github.com/yakaz/elasticsearch-action-updatebyquery/ > Just set a match_all query. > > Bulk API with updates to all the docum

Re: Need help/suggestion for the massive queries user case

Ok. you are right. I was able to set the maximum heap size and now it looks like: "jvm" : { "timestamp" : 1409684394790, "uptime_in_millis" : 128401, "mem" : { "heap_used_in_bytes" : 520385872, "heap_used_percent" : 24, "heap_committed_in_bytes" : 2138767360, "heap_max_in_bytes" : 21387673

Re: Need help/suggestion for the massive queries user case

I think your JVM settings are incorrect as we can see with heap_max -- David ;-) Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 2 sept. 2014 à 20:37, Yuheng Du a écrit : BTW, the node stats info shows the jvm usage info as follows: "jvm" : { "timestamp" : 1409682903048, "uptime_in_mi

Re: Need help/suggestion for the massive queries user case

BTW, the node stats info shows the jvm usage info as follows: "jvm" : { "timestamp" : 1409682903048, "uptime_in_millis" : 488996, "mem" : { "heap_used_in_bytes" : 48476472, "heap_used_percent" : 9, "heap_committed_in_bytes" : 73412608, "heap_max_in_bytes" : 515571712, "non_heap_used_in_bytes" : 570

Re: Need help/suggestion for the massive queries user case

Ok. The shards are recovered now. I tried to set the ES_HEAP_SIZE to 2g, but I am still getting "Nodes not available" error: What can be the cause then? Thanks! Yuheng On Tuesday, September

Re: Transport Client connectedNodes() duplicates

Interesting… for what it’s worth, it was indeed binding to the local IPv6 wildcard address, but publishing the external IPv4 address: [2014-09-01 21:14:05,560][INFO ][http                     ] [Diablo] bound_address {inet[/0:0:0:0:0:0:0:0%0:9200]}, publish_address {inet[/192.168.1.5:9200]}

Re: writing a custom scoring plugin

Thanks Vineeth. I stumbled upon that example too, thats one of the best resources I have found so far. Thanks Joerg, this example looks very interesting to me. I see that you are sending in the extra parameters with the query, but i would like the plugin to read the data from external source and n

Re: aggregation of hierchical elements possible?

Helli Vineeth, thx for your response. Your proposal 1. seems to be similar to the path-tokenizer, which I used, isn't it? "settings" : { "index" : { "analysis" : { "analyzer" : { "path-analyzer" : { "type" : "custom"

Re: EL setup for fulltext search

Hard to say without looking at your query, but perhaps you are experiencing query parser issues. The query string query uses the standard query parser, which might does not tokenize terms in the way your custom tokenizer might. Try using match queries, which does not use the query parser to see if

Re: Transport Client connectedNodes() duplicates

The two nodes are okay. The #transport# node connection on Mac OS X is just using IPv6. ES enumerates all network devices so it finds out about that connection. No need to be concerned, it can be configured if you want that, either by JVM property, or by elasticsearch settings in config for the net

Re: writing a custom scoring plugin

You can write a function score plugin. Here is an example: https://github.com/jprante/elasticsearch-functionscore-conditionalboost Jörg On Tue, Sep 2, 2014 at 6:48 PM, Srinivasan Ramaswamy wrote: > I am trying to write a custom scoring plugin that will let me provide some > extra data that

Re: Add field to all documents of an index.

Hello John , Update by query plugin would be a good choice - https://github.com/yakaz/elasticsearch-action-updatebyquery/ Just set a match_all query. Bulk API with updates to all the documents should also work. Thanks Vineeth On Tue, Sep 2, 2014 at 7:01 PM, John Smith wrote: > Hi I

Re: Mapping issue?

Hello Seb , Declaring it as not_analyzed should do the trick. Can you paste the mapping you have applied. Thanks Vineeth On Tue, Sep 2, 2014 at 8:20 PM, Seb Emin wrote: > I'm storing search terms in ElasticSearch, and then trying to do things > like find the top 5 search terms. > > W

Re: writing a custom scoring plugin

Hello Srinivasan , You cna easily create this as a plugin. There are many examples here - https://github.com/imotov/elasticsearch-native-script-example/tree/master/src/main/java/org/elasticsearch/examples/nativescript/script Thanks Vineeth On Tue, Sep 2, 2014 at 10:18 PM, Srinivasan R

writing a custom scoring plugin

I am trying to write a custom scoring plugin that will let me provide some extra data that is not in the index (through an external service) and do customized ranking. I would ideally like to custom score all the documents in the index that match the query. Currently I am planning to implement

Re: IDF per category

Hello , You can do a 2 level aggregation term query with first level as category and second level as the content. { "aggregations": { "categories": { "terms": { "field": "category" }, "aggregations": { "categories": { "terms": { "fiel

Re: New kibana dashboards added

I would definitely use the logstash configs if you posted them. We have been using Es for a while for a product search and are getting into the early stages of looking at logstash + ES for analytics. Those configs would definitely kickstart our efforts. Thanks, -Kelly On 08/20/2014 07:41 AM

Re: Sense on github abandoned?

Ok, so here is what I wanted to suggest: every query could have a "Validate & explain" link in the little dropdown: Just under "Auto indent". And as you

Re: Sense on github abandoned?

> About Sense, it feel like a closed source software now. Yes, Sense is now part of Marvel, which is closed source. > Can we have an open bug tracker to suggest idea? We currently collect ideas and issues using this group. Would love to hear it if you have a suggestion! Cheers, Boaz On Tue,

Re: Sense on github abandoned?

About Sense, it feel like a closed source software now. Can we have an open bug tracker to suggest idea? or even better, open the code to accept pull requests? Also, this is a great example of why one should always set a license when publishing code :P Damien On Wednesday, April 30, 2014 10:

Re: ELK stack Implementation on Windows using nxlog - Help required

Hi , Thanks for your reply .. Here after i will not attach the documents. I am checking the ES plugins and let you know . -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send

IDF per category

I have a set of categories containing some products I would like to index the label and the category of the products (to handle record linkage). the issues is: some terms are common for some categories but could be discrimant for other categories. So I would like to compute the IDF of a word

Re: [Hadoop] Hadoop plugin indices with ':' colon - not able to snapshot (?)

Thanks. Yes, will try to dig around the plugin code. Not entirely sure I get how generic this is, my understanding was that's it was more HDFS-specific. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop r

Mapping issue?

I'm storing search terms in ElasticSearch, and then trying to do things like find the top 5 search terms. When I write queries to do this, instead of getting top 5 occurring search terms, I'm getting the top 5 occurring words. For example: term1 = Tomb Raider: Temple of Light term2 = Tomb Raide

Re: [Hadoop] Hadoop plugin indices with ':' colon - not able to snapshot (?)

If the alias method doesn't work for you then escaping _might_ be the only way (assuming there's going to be a reliable way to apply it) - I suggest raising an issue since this sounds like a generic functionality that could work across all snapshot/restore plugins. Cheers, On 9/2/14 4:03 PM, M

Re: Geo bounds aggregation as a sub aggregator?

Ok, yes this does look like a bug so if you could raise an issue on github and post the issue link back in this thread, that would be great. Thanks Colin On Tuesday, 2 September 2014 15:23:23 UTC+1, Owain Braddick wrote: > > Hi, > > I hit exactly the same issue today. It seems that the problem

Re: Copy Indices to new Server

Hello Andrew , Did you copy the data folder or do a snapshot/restore - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html#_snapshot Thanks Vineeth On Tue, Sep 2, 2014 at 7:05 PM, Andrew Lakes wrote: > Hey guys, > > we have set up a new physi

Re: Need help/suggestion for the massive queries user case

You have to start again the second node on the same machine to get the other shards back. Then when you cluster is yellow again, you can switch off the second node. BTW, you should set ES_HEAP_SIZE instead of dealing with Xmx Xms (but this will do the same thing behind the scene). See  http://w

Re: Geo bounds aggregation as a sub aggregator?

Hi, I hit exactly the same issue today. It seems that the problem occurs when the number of buckets in the outer geohash grid aggregation goes above a certain number (around 50 in my tests but that could be system dependent). The stack trace is below but doesn't tell much. If someone can confir

Re: Need help/suggestion for the massive queries user case

I tired 2g, It only worked when I use ./elasticsearch -Xmx2g -Xms2g. Currently some of my shards are stated as "unassigned" in my kopf web interface. It happened after I started 3 nodes at the same time in my desktop yesterday. There should be 3G of data in the cluster, but know I see only 1.92

Re: Need help/suggestion for the massive queries user case

Try 2g -- David ;-) Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > Le 2 sept. 2014 à 15:49, Yuheng Du a écrit : > > I tried to run ./elasticsearch --node.name=esmaster -Xmx=2G -Xms=2G. > But it gives me the error msg: Invalid maximum heap size: -Xmx=2G, > > How can I deal with this?

Re: Need help/suggestion for the massive queries user case

I tried to run ./elasticsearch --node.name=esmaster -Xmx=2G -Xms=2G. But it gives me the error msg: Invalid maximum heap size: -Xmx=2G, How can I deal with this? Thanks On Tuesday, September 2, 2014 3:46:26 AM UTC-4, David Pilato wrote: > > Ha! On your desktop? So that's not a production issue.

Copy Indices to new Server

Hey guys, we have set up a new physical ES-server in order to get more ressources for our ELK-Stack. On our old server we got about 1.8 TB/ 127 Indices which have to be available on the new server. So we copied the the indices to the new server, which have the same configuration as the old se

Add field to all documents of an index.

Hi I need to add a new field (with a default) to all documents already indexed. I guess bulk update will do it? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email

Re: [Hadoop] Hadoop plugin indices with ':' colon - not able to snapshot (?)

I'm afraid it's the other way around, i.e. we already have 50 indices or so (+tools) using colon in the base name. And just to check that as well, specifying alias in 'indices' instead of the base name still results in the same error. -- You received this message because you are subscribed t

Re: Using elasticSearch as repository for UDP published Ceilometer data through logstash and exception is being thrown 'invalid version format'

Bump... Anyone??? On Friday, August 29, 2014 11:28:01 AM UTC-4, Brian Callanan wrote: > > > Hi, Need a little help. I'm Using Openstack Ceilometer and I've configured > it to push metered data over UDP to a host:port. I installed logstash and > configured it to receive the the UDP data from Cei

Re: aggregation of hierchical elements possible?

Hello Markus , I cant seem to think of any straight method , but then you can try the following 1. Apply source transform script to convert /a/b/c => [ /a , /a/b , /a/b/c ] - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-transform.html#mapping-transform

Re: [Hadoop] Hadoop plugin indices with ':' colon - not able to snapshot (?)

If it's only the index that you are interested, why not use aliases instead? Have the index snapshot/restored through a legal fs name (crawl-1) and have an alias that uses the special character (crawl:1) pointing to it? On 9/2/14 2:20 PM, Mateusz Kaczynski wrote: Hi Costin, thanks for the respo

Re: [Hadoop] Hadoop plugin indices with ':' colon - not able to snapshot (?)

Hi Costin, thanks for the response. Yes, I understand that this is a restricted character and would require escaping. There are perhaps 2 separate issues here: 1) If 'indices' is not specified, i.e. curl -XPUT "localhost:9200/_snapshot/hdfs-cluster/snapshot_1" elasticsearch is going to complai

Re: [Hadoop][pig] How to set the document id in ESStorage?

You can through the 'es.mapping.id' parameter; see [1] Cheers, [1] http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/configuration.html#_mapping On 9/2/14 11:24 AM, Arokia Daniel Arokiadoss wrote: Hi, I've been doing some Aggregation functional using the ES STORE and Hadoop-

Re: [Hadoop] Setting Document ID in Map Reduce Mapper

Actually the issue you refer to was not regarding a bug but rather a feature that the user was not aware of - hence the closing comment pointing to the documentation. In general, when in doubt, feel free to open an issue, even if it's a duplicate. Glad to hear the problem has been sorted out.

Re: [Hadoop] Setting Document ID in Map Reduce Mapper

Indeed updating to 2.0.1 solves the problem. I looked for issues in github related with specifying a document id in MapReduce and the one [1] I found was closed without explanation and a link to the online documentation. So I wrote here because I thought it wasn't a bug. I thought it was a m

Re: How to Speed Up Indexing

Hi, mohan My lastest testing, indixing data about 14000 documents. 1. Tuning BulkAPI params, 6m. Before tuning, time is 14m. [INFO] Total time: 6:06.173s [INFO] Finished at: Tue Sep 02 15:40:36 CST 2014 [INFO] Final Memory: 27M/312M ref: http://www.elasticsearch.org/guide/en/elasticsearch/re

Re: Get distinct data

Hello Alex , Aggregate based on sourceId field and you should get distinct sourceId values. Thanks Vineeth On Tue, Sep 2, 2014 at 12:48 PM, Alex T wrote: > Hi Vineeth! Thanks for your answer. > > I use term aggregation, but I get anyway response with 2 documents, > response data for

Re: How to Speed Up Indexing

Hello Haiwei , The more hardware you can get , it should be better unless the data is too small. So if there are 10 machines , set the shards as 10 , so that the index can uniformly use all the resources. Thanks Vineeth On Tue, Sep 2, 2014 at 12:13 PM, wrote: > Hi, > > "I

Query and filter improvement not available ?

Hi all I successfully deployed KIBANA 3.1.0 and I made a first dashboard based on ELK stack to monitor log files. However I m quiet surprise since my KIBANA GUI does not look those shown in on screenshots available on the ElasticSearch web site : Query & Filter (on top of dashboard) are absolut

[Hadoop][pig] How to set the document id in ESStorage?

Hi, I've been doing some Aggregation functional using the ES STORE and Hadoop-PIG functionality . I want to store the aggregated data in Elasticsearch through PIG. While storing the data , elasticsearch generates document id automatically. Is there a way to set document id manually in run time

Re: Need help/suggestion for the massive queries user case

Ha! On your desktop? So that's not a production issue. In that case, you should may be give more memory to all your elasticsearch nodes. What is your current HEAP size? --  David Pilato | Technical Advocate | Elasticsearch.com @dadoonet | @elasticsearchfr Le 2 septembre 2014 à 09:17:31, Yuheng

Re: Get distinct data

Hi Vineeth! Thanks for your answer. I use term aggregation, but I get anyway response with 2 documents, response data for example: { "took":23, "timed_out":false, "_shards":{"total":5,"successful":5,"failed":0}, "hits":{ "total":2, "max_score":null, "hits":[ { "_index":"feedit

Re: Need help/suggestion for the massive queries user case

Each query is doing a search in _all indexes in the cluster. Maybe that's also a hold up of the availability of ES server. I tried to run 3 ES nodes on my desktop machine but it does not seems to help, it also crashed after some time. On Tuesday, September 2, 2014 3:10:13 AM UTC-4, David Pilat

Re: Need help/suggestion for the massive queries user case

May be you could increase the number of replicas and nodes so your cluster could handle more queries at a time. The main cost is of course in term of machine and disk space. Can't tell really more without more details about what you are doing exactly. May be others have some ideas? --  David Pi