Re: My elasticsearch powered grafana server gives me "org.apache.lucene.index.CorruptIndexException" exceptions when I try to restore from backup. Help!

Grafana uses graphite, do you mean Kibana? On 1 May 2015 at 15:09, jblock wrote: > Had a problem with my grafana server today so I am now forced to restore > from backups. When I try to restore from any of my many backups, I get > exceptions such as this: > > Caused by: org.apache.lucene.index.C

My elasticsearch powered grafana server gives me "org.apache.lucene.index.CorruptIndexException" exceptions when I try to restore from backup. Help!

Had a problem with my grafana server today so I am now forced to restore from backups. When I try to restore from any of my many backups, I get exceptions such as this: Caused by: org.apache.lucene.index.CorruptIndexException: [][1] Corrupted index [corrupted_DOarE0krSua6FJ46T5dFBQ] caused by:

Re: How to replicate this type of search

Although the syntax is straight Lucene (query string query), I suspect that Github and other sites parse the query term to create a format similar to the one John mentioned. Cheers, Ivan On May 1, 2015 1:22 AM, "Peter Sorensen" wrote: > Sorry for the vague title. If I knew what to call what I w

Re: How to replicate this type of search

Hey Peter, They're likely using a filtered query . Maybe on a "status" field for the "is:open" and a "type" field for the "is:issue". It could look something like this: { "query":{ "filtered":{

Re: How to Boost

I was able to get this to work in ES using head and got back what i needed. However translating to ElasticSearch.net has been an issue. I have an anonymous type doing a match all query and this works fine and dandy. I am trying to do the multi_match as above but it isn't returning any results

startup error

When I attempt to start logstash and test a config file I get this: c:\Logstash\logstash-1.5.0-rc3\bin>logstash -t testConfig.conf --verbose io/console not supported; tty will not be manipulated Error: Usage: c:\Logstash\logstash-1.5.0-rc3\lib\logstash\runner.rb [OPTIONS] has anyone else seen

How to take a snapshots of a specific index with the php library?

When I try to list a specific index to take a snapshot of, the library seems to ignore it, and instead takes a snapshot of my entire cluster. It's the end of the day, and I'm likely missing something obvious, so any help would be appreciated. Ultimately, I'm trying to do the equivalent of: cur

How to replicate this type of search

Sorry for the vague title. If I knew what to call what I was looking for, I'd have a much easier time finding it! Anyways, I often see sites using filters right in the query box. For instance, on Github, you can see open issues by typing ` is:open: is:issue {search term} ` What element of ES i

Re: Help on using Elastic Search Api from SPark

"It looks like it is related to an intellij maven import issue as it is not downloading any new dependency I am adding." Pardon? I don't understand what you are saying. Are we talking about the same project? Have you looked at the docs I mentioned? In particular [1] the install page? I ask since

Re: Kibana 4 initial configuration doesn't pick up the expected indices

Thanks. I must be missing the meaning of this setting. I tried both "*" and "*.MM.DD" and still don't get any matching indices. On Monday, 27 April 2015 13:04:51 UTC+10, Mark Walkom wrote: > > You cannot use wild card prefixes. > > There is functionally no difference between * and *-something,

Re: Help on using Elastic Search Api from SPark

It looks like it is related to an intellij maven import issue as it is not downloading any new dependency I am adding. On Thu, Apr 30, 2015 at 6:36 PM, Deepak Subhramanian wrote: > Hi , > > I am trying to use Elastic Search Rest Api for searching Elastic > Search Index from Spark Code in Scala. I

Re: How to set Marvel license programmatically in Ansible deploy?

Oh well. At least I now have a definitive answer. Thanks! -joel On Thursday, April 30, 2015 at 5:37:40 PM UTC-4, Mark Walkom wrote: > > Unfortunately there isn't a way to do this, but I've raised a feature > request to get this added. > > On 30 April 2015 at 23:39, Joel Potischman > wrote: > >>

Re: JDBC River missing documents??

wow, thanks for sharing! Best, Jörg On Thu, Apr 30, 2015 at 10:43 PM, GWired wrote: > The below will build a table in SQL to store Refresh times. The first > time it runs it will put in an entry and going backwards in time until all > records are retrieved. Once compete it will retrieve bas

Re: ElasticSearch built-in Jackson stream parser is fastest way to extract fields

Swati, Well, I tend not to use the built-in Jackson parser anymore. The only advantage I've seen to stream parsing is that I can dynamically adapt to different objects in my own code. But I can't release the code since it's owned by my employer. And for most tasks these days, I use the Jackson

Re: How to set Marvel license programmatically in Ansible deploy?

Just to add, the shield repo isn't publically accessible do to it being one of our commercial plugins, so I can't link you to the issue sorry! On 1 May 2015 at 07:37, Mark Walkom wrote: > Unfortunately there isn't a way to do this, but I've raised a feature > request to get this added. > > On 30

Re: How to set Marvel license programmatically in Ansible deploy?

Unfortunately there isn't a way to do this, but I've raised a feature request to get this added. On 30 April 2015 at 23:39, Joel Potischman wrote: > We have a license for Marvel and while the documentation just says to > enter it through the Marvel UI when prompted, we script 100% of our > infra

Re: Failed to get setting group for [threadpool.] setting prefix and setting [threadpool.bulk] because of a missing '.'

How are you setting this? On 30 April 2015 at 22:02, wrote: > Hello there > > I noticed the following warning message in my nodes logs when starting: > > [2015-04-30 12:40:13,265][INFO ][node ] [***server***] > started > [2015-04-30 12:40:32,154][WARN ][node.settings

search query not returning

Hello, Over the past two days I've had the issue where all of a sudden I'm unable to issue a search query to all or certain indexes. I'm able to issue a basic _mapping query to each index and it returns just fine. But if I try a search query of any kind it never returns. Disk space, memory and

Re: Kibana - Is there such a thing as a "permanent filter"?

Not at this stage, this is something that is being worked on for future releases. On 1 May 2015 at 03:22, Andrew Ruslander wrote: > I have a system that multiple users use. I want to make a single Kibana > dashboard that the users could pull up, but it would only show them their > own data, not

Re: Marvel showing unresponsive nodes but active data

Check time is syncd between your nodes. On 1 May 2015 at 05:23, Tristan Hammond wrote: > Hmm... > > Digging into this a little, it seems like it's an issue specifically w/ > the Master node. If I load up Marvel on any of the other nodes, they see > everything but the Master (and the updated inde

Re: Field-length norm fails on fields with 3 and 4 words

The field norm is computed at index time and is stored in a single byte, which can lead to a loss in precision. This behavior might have changed with newer versions of Lucene, but probably not. Ivan On Apr 30, 2015 6:42 PM, "Fil ES" wrote: > Hello, > > I am experiencing an very annoying behaviou

Multi word query - Highlight should show all matched words instead of single word

I am using a multi word search query used for to find a match for text that user is typing. (Im not using Suggesters) eg. If a doc has a field with the text "... ... ... .. The Moon is a part... .. moon is abc ... .. universe is xyz... .. . galaxy in the universe ." The user t

Re: JDBC River missing documents??

The below will build a table in SQL to store Refresh times. The first time it runs it will put in an entry and going backwards in time until all records are retrieved. Once compete it will retrieve based on fields lastmodifieddate and createddate. The time based chunks allow ES enough time t

Re: ElasticSearch built-in Jackson stream parser is fastest way to extract fields

Hi Brian, I am new to the elasticSearch and currently working with version 1.5.1. I am working on parsing a json string returned by getSourceAsString() method into an object. JsonParser class in elasticsearch (import org.elasticsearch.common.jackson.core.JsonParser) doesn't allow me to create

Re: es spark with local elasticsearch listening on non default ports

Since the RDD is created lazily, there's no validation done during its declaration - the actual logic behind it is executed when an operation (like count) is performed. It looks like the nodes are not properly accessed; potentially the port is not being picked up. Can you try passing the es.node co

Re: Marvel showing unresponsive nodes but active data

Hmm... Digging into this a little, it seems like it's an issue specifically w/ the Master node. If I load up Marvel on any of the other nodes, they see everything but the Master (and the updated index data it would hold). Cluster still reports green w/ all 5 nodes for some reason, though. It'

Re: es spark ignoring fields query parameter

Please raise an issue (more info about what's needed can be found here [1]). Cheers, [1] http://www.elastic.co/guide/en/elasticsearch/hadoop/master/troubleshooting.html#help On Thu, Apr 30, 2015 at 6:28 PM, Israel Klein wrote: > Hi Costin, > > > It was my mistake in the question. First I have

Re: Help on using Elastic Search Api from SPark

Take a look at Elasticsearch Hadoop project [1]. It features integration with various Hadoop libraries including Apache Spark [2]. Cheers, [1] https://www.elastic.co/products/hadoop [2] http://www.elastic.co/guide/en/elasticsearch/hadoop/current/spark.html On Thu, Apr 30, 2015 at 8:36 PM, Deepak

How to build faceted columns on the left nav on Discover page

How do i add faceted columns on the left on the Discovery page? I want to get fields like 'Destination ip' and 'Destination host' and their count. Is this possible in Kibana 4? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe fr

Re: possible networking problem?

You're right, I found this problem about an hour ago. There's a mistake in a doc which I where I picked it up from. Thanks. On Thu, Apr 30, 2015 at 9:41 AM, Christian Dahlqvist wrote: > Hi, > > I think there is some confusion about the port number used. Kibana 4 by > default listens to port 56

Re: In which case ElasticSearch will return 429 ?

Hi, As explained in the blog post, increasing the queue size will not improve performance, just make you store more data in memory on the cluster awaiting processing. It could actually instead end up reducing performance. It looks like you are hitting the limit of your cluster and that the numb

java recovery api, stale shard information?

Hi, I am using ES 1.5.2 For a given index I am trying to determine which ES node(s) hold the primary shards at any given time. I am using the Java API I make a RecoveryRequest for a specific index to a node, and get back a RecoveryResponse, I then do the following to create a simple list that

Help on using Elastic Search Api from SPark

Hi , I am trying to use Elastic Search Rest Api for searching Elastic Search Index from Spark Code in Scala. I tried wabisabi and searchboxio . For some reason I am not able to import the classes even though I added the packages in my pom . I am using Spark 1.3.0. Is there any other package I can

Kibana - Is there such a thing as a "permanent filter"?

I have a system that multiple users use. I want to make a single Kibana dashboard that the users could pull up, but it would only show them their own data, not anyone elses. I think I can do this by using the templated dashboards and establishing a filter that would take their username in the

Re: Documents not deleted when using DeleteRequest within BulkProcessor

Thank you for the reply, David. We are using an alias to delete. Is that a problem? Indexing with the alias through the bulk processor works fine. There are no errors reported, it just seems to disappear into the oblivion. Here's our code for the BulkProcessor: public static BulkProcessor g

Field-length norm fails on fields with 3 and 4 words

Hello, I am experiencing an very annoying behaviour of the elastic search score calculating algorithm - the field length fails to find a difference between fields which contain 3 and 4 words. Always return same score for both. Example: LANCA HOTEL EXTREME and MASSIVE AMAZING HOTEL GROUP wou

Re: possible networking problem?

Hi, I think there is some confusion about the port number used. Kibana 4 by default listens to port 5601, which based on the output sample you provided seems to not have been changed. In all your examples you are however looking for port 5061, not 5601. Can you check if you are able to connect

Re: possible networking problem?

Yes, Kibana is out there listening on port 5061. On Wed, Apr 29, 2015 at 9:59 PM, David Pilato wrote: > If you are using all defaults settings, checked windows firewall and tried > http://0.0.0.0:5601/ (and not 127.0.0.1), I don't see what is happening. > Is Kibana still running? > > David > > L

Re: Splunk vs. Elastic search performance?

Hi everyone, Recently I use Splunk Enterprise for my company's need and I am new to Elasticsearch (and other open-source alternatives mentioned here). But I am also interested in this analysis (and focus group) about these two solutions. One of the first analysis I read in Web is from *riskfocu

[ElasticSearch-Spark] [SQL] Passing pushed down filters to elastic search

While reading through the source of ES Spark integration, I realized that integration doesn't make use of SQL filter pushdowns. I have made an initial attempt at making it work and things look pretty encouraging. I have open a github issue (https://github.com/elastic/elasticsearch-hadoop/issues

es spark with local elasticsearch listening on non default ports

Hi, I am using es-spark_2.11-2.1.0.Beta4 on Spark 1.3.1 shell against elasticsearch 1.4.4. I have elasticsearch settings: http.port: 9201 transport.tcp.port: 9301 I am setting the port in conf/spark-defaults.conf: spark.es.port 9201 creating an esRdd works fine, however, trying an operation

Re: es spark ignoring fields query parameter

Hi Costin, It was my mistake in the question. First I have tried the above and it did not work at all(obviously) but then I have tried val esRdd = sc.esRDD("my_index/my_type"," ?fields=field_name") as well with the mentioned above result. On Thursday, April 30, 2015 at 5:47:15 PM UTC+3, Co

[ANN] Elasticsearch Smart Chinese Analysis plugin 2.4.4 released

Heya, We are pleased to announce the release of the Elasticsearch Smart Chinese Analysis plugin, version 2.4.4. Smart Chinese Analysis plugin integrates Lucene Smart Chinese analysis module into elasticsearch.. https://github.com/elasticsearch/elasticsearch-analysis-smartcn/

Re: es spark ignoring fields query parameter

To specify the query, use es.query param or the second string param in esRDD. The doc provide detailed information. In a future version, we'll throw an exception if a query is specified in the index. On Apr 30, 2015 5:03 PM, "Israel Klein" wrote: > Hi, > > I am using es-spark_2.11-2.1.0.Beta4 on

[ANN] Elasticsearch Phonetic Analysis plugin 2.4.3 released

Heya, We are pleased to announce the release of the Elasticsearch Phonetic Analysis plugin, version 2.4.3. The Phonetic Analysis plugin integrates phonetic token filter analysis with elasticsearch.. https://github.com/elasticsearch/elasticsearch-analysis-phonetic/ Release Notes - elasticsear

[ANN] Elasticsearch Stempel (Polish) Analysis plugin 2.4.3 released

Heya, We are pleased to announce the release of the Elasticsearch Stempel (Polish) Analysis plugin, version 2.4.3. The Stempel (Polish) Analysis plugin integrates Lucene stempel (polish) analysis module into elasticsearch.. https://github.com/elasticsearch/elasticsearch-analysis-stempel/ Rel

Re: [ANN] Elasticsearch Japanese (kuromoji) Analysis plugin 2.4.3 released

Sorry. Was a typo. You should have read Update to elasticsearch 1.4.5 Le jeudi 30 avril 2015 15:48:44 UTC+2, David Pilato a écrit : > > Heya, > > We are pleased to announce the release of the *Elasticsearch Japanese > (kuromoji) Analysis plugin*, *version 2.4.3* > > The Japanese (kuromoji) Anal

[ANN] Elasticsearch ICU Analysis plugin 2.4.3 released

Heya, We are pleased to announce the release of the Elasticsearch ICU Analysis plugin, version 2.4.3. The ICU Analysis plugin integrates Lucene ICU module into elasticsearch, adding ICU relates analysis components.. https://github.com/elasticsearch/elasticsearch-analysis-icu/ Release Notes -

es spark ignoring fields query parameter

Hi, I am using es-spark_2.11-2.1.0.Beta4 on Spark 1.3.1 shell against elasticsearch 1.4.4. Doing: import org.elasticsearch.spark._ val esRdd = sc.esRDD("my_index/my_type?q=fields:field_name") esRdd.take(1) results in getting the whole document source and not just the specified field. Than

[ANN] Elasticsearch Japanese (kuromoji) Analysis plugin 2.4.3 released

Heya, We are pleased to announce the release of the Elasticsearch Japanese (kuromoji) Analysis plugin, version 2.4.3. The Japanese (kuromoji) Analysis plugin integrates Lucene kuromoji analysis module into elasticsearch.. https://github.com/elasticsearch/elasticsearch-analysis-kuromoji/ Rele

How to set Marvel license programmatically in Ansible deploy?

We have a license for Marvel and while the documentation just says to enter it through the Marvel UI when prompted, we script 100% of our infrastructure creation and deployments (Ansible). My goal is to be able to start with a bare VM, run our deployment, and when the dust settles, our producti

Re: LIKE Query

You need first to understand that a search engine is not a SQL like system. It’s based on a inverted index so what you query is not your source data but the index you have created. The way you build your index is extremely important then. You could imagine using wildcard queries but first it woul

Re: Troubleshooting Elasticsearch queries

I think you said that it returns 1.5M records because you saw that in total_hits field, right? This is because you did write any query but only a facet. So the resultset is the full dataset for this index. I would write something like this: https://gist.github.com/dadoonet/0b161423d0acafecb012

Trying to set not_analyzed to a field

I have a setup where I am shipping windows eventlogs using nxlog in JSON format towards logstash which gets put into an elasticsearch cluster. The "Message" field is currently being analysed which means that if I want to visualise on Kibana all events sorted by Message the Message field gets sp

LIKE Query

Hi All, Is there any way to find documents by querying fields for matching terms similar to database LIKE I have gone thru some of the queries which are dated 2012, 2011. None of them has straightforward answer. Not sure of we have the feature in current version 1.4.5 I want some thing like ...

Troubleshooting Elasticsearch queries

Hi, I've included my gist here https://gist.github.com/trekr5/db0b0f577cab10d95257. I'm trying to extract the number of status 500 errors from an ElasticSearch server over a 2 hour period. The problem is that the query returns too many records. I should be receiving 272 errors over a 2 hour ti

Failed to get setting group for [threadpool.] setting prefix and setting [threadpool.bulk] because of a missing '.'

Hello there I noticed the following warning message in my nodes logs when starting: [2015-04-30 12:40:13,265][INFO ][node ] [***server***] started [2015-04-30 12:40:32,154][WARN ][node.settings] [***server***] failed to refresh settings for [org.elasticsearch.th

update dynamically set mapping using PUT mapping API

Hey everyone , I have stucked over an issue. I have an index with the following mapping currently : "mappings": { "product": { "dynamic_templates": [ { "template_name": { "mapping": { "type": "strin

sequentials filters

Hi, some Analytics Tools support "segements" which are basically nothing more than must, should or must_not filters. Something more advanced is a "then" option: http://blogs.adobe.com/digitalmarketing/wp-content/uploads/2014/07/fallout_segment_02.png Has someone an idea how to implement this i

Re: How to fix cluster health yellow with Elastic Search

Ok i understand. Thanks Pilato On Thursday, April 30, 2015 at 3:38:47 PM UTC+5, David Pilato wrote: > > As I said, it’s not an issue. Your cluster is working well. > But if you want to allocate one replica for each primary shard, you will > need one more node. > If you set number of replica to 2,

Re: The best way to migrate data from 3 indices to new index

Hi Mark, I'm trying to use javascript to do that by bulk operation but I found some issue when source data contain "\\" that cannot insert to target index that show below error MapperParsingException[failed to parse [summary]]; nested: JsonParseException[Unrecognized character escape 's' (code

Re: [ANN] Elasticsearch Mapper Attachment plugin 2.3.0 released

Ok, So handling at the client side would be the best solution for the same. Yes, we do have records in TB's which may lead to billions of documents to be reindexed or handled differently. -- View this message in context: http://elasticsearch-users.115913.n3.nabble.com/ANN-Elasticsearch-Mapper-

Re: [ANN] Elasticsearch Mapper Attachment plugin 2.3.0 released

reindex is not a feature in elasticsearch nowadays. You can read http://www.elastic.co/guide/en/elasticsearch/guide/master/reindex.html How many documents you have to reindex? IIRC your case you could have a lot, right? --

Re: [ANN] Elasticsearch Mapper Attachment plugin 2.3.0 released

Yes, Though that can be handled as an exception in our application. But I was just looking for as if any thing is there in ES which can re index old document in parallel, and make the content present in older records under "_content" only instead of "content". -- View this message in context

Re: [ANN] Elasticsearch Mapper Attachment plugin 2.3.0 released

So why not doing that in your application? If you look into _source.file.content and _source.file._content, older docs will have BASE64 content in content and null in _content and the opposite for newer docs. Do I miss anything? -- David Pilato - Developer | Evangelist elastic.co @dadoonet <

Re: How to fix cluster health yellow with Elastic Search

As I said, it’s not an issue. Your cluster is working well. But if you want to allocate one replica for each primary shard, you will need one more node. If you set number of replica to 2, your cluster will be yellow again until you start a new node. If you really want to change the node name, yo

Re: [ANN] Elasticsearch Mapper Attachment plugin 2.3.0 released

Hi , Thanks for a quick response, Yes , I am making a change content to _content as an index request. Here my worry is like our application is already deployed , so with new changes being done I will not be able to fetch the older records using the same application as older records will be return

Re: In which case ElasticSearch will return 429 ?

Hi, Thx for your answer!! However, I still feel confuse because when we keep 200 concurrent thread cyclically send requests (Indexing request) to cluster，it performence good and we guess there still have remaining performance space.You can see the "Throughput" is 5532.8/sec.

Re: [ANN] Elasticsearch Mapper Attachment plugin 2.3.0 released

Just a note before answering, the change is about the document you send to elasticsearch. Instead of sending: { "file": { "content": "VGhpcyBpcyBhbiBlbGFzdGljc2VhcmNoIG1hcHBlciBhdHRhY2htZW50IHRlc3Qu", "_name": "myfilename.txt" } } You now send { "file": { "_content": "VGhpcy

Re: [ANN] Elasticsearch Mapper Attachment plugin 2.3.0 released

Hi dadoonet, Currently we were using Elasticsearch 1.3.0 and supported Mapper Plugin for the same. Now we are moving Elasticsearch 1.5.2 so we need to have the updated mapper att plugin as well. As in the latest one content is deprecated by _content, so in our application we need to change the co

Re: How to fix cluster health yellow with Elastic Search

Thanks for your quick reply. Two nodes is necessary for this issue? And Can i define two nodes in single config file like: node: name: firstnode name: secondnode ? and just start by bin/elasticsearch On Thursday, April 30, 2015 at 2:12:22 PM UTC+5, David Pilato wrote: > > First this is not a

Re: How to fix cluster health yellow with Elastic Search

First this is not a problem. Just showing you that all primaries are allocated but nor replicas. How to solve it? Increase the number of nodes. Obviously elasticsearch doesn’t replicate on the same node or Reduce the number of replicas (you can do it live) or Live with it Now coming back to th

How to fix cluster health yellow with Elastic Search

Hello.! Elastic search stuck in yellow status. I searched a lot about this problem and i find out same answer that to create two nodes. So i create two nodes by creating two config file elasticsearch.yml and elasticsearch-1.yml and start by command: elasticsearch -Des.config=d:/elastics

Re: In which case ElasticSearch will return 429 ?

Hi, It means that Elasticsearch is not able to keep up with the number of requests you are sending and that the internal queues are filling up. A discussion around this can be found in the following blog post: https://www.elastic.co/blog/performance-considerations-elasticsearch-indexing Best r

Re: Function score query fails with field_value_factor on not (yet) existing field

The error originates from FieldValueFactorFunctionParser.java (line 89), in FieldValueFactorFunc

In which case ElasticSearch will return 429 ?

When doing pressure test on ES cluster，it sometimes may return 429(too many request). I can't find any detail docs about this.In which case ElasticSearch may return 429 ? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from t

Re: More memory or more CPU cores help better performance?

Thanks a lot Jörg! On Thursday, April 30, 2015 at 3:13:36 PM UTC+8, Jörg Prante wrote: > > As said, it depends. > > When bulk-indexing documents, for example, my multi-threaded workload is > network-bound. It can easily be made CPU-bound by pre-processing documents > in single thread mode. Certa

Re: More memory or more CPU cores help better performance?

As said, it depends. When bulk-indexing documents, for example, my multi-threaded workload is network-bound. It can easily be made CPU-bound by pre-processing documents in single thread mode. Certain queries are CPU-bound, others not. If I retrieve millions of documents in a row, decompression ove