Re: elasticsearch curator — version 1.1.0 released

I use curator for my logstash logs, which are not really used since my team prefers our other logging solution (which is far worse, but that is what they know). I use snapshots for our main Elasticsearch cluster, so I should start use curator for the snapshots. Nothing about restore though. -- Iv

Re: elasticsearch curator — version 1.1.0 released

It has a prefix setting, but not a suffix. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 14 June 2014 13:35, Ivan Brusic wrote: > The addition of the snapshot feature is interesting, but I just wish there > was a

Re: elasticsearch curator — version 1.1.0 released

The addition of the snapshot feature is interesting, but I just wish there was a way to specify the index names instead of just specifying the dates. I haven't downloading it yet, but it does have a prefix setting. I need a suffix setting. -- Ivan On Fri, Jun 13, 2014 at 5:38 PM, Mark Walkom w

elasticsearch curator — version 1.1.0 released

http://www.elasticsearch.org/blog/elasticsearch-curator-version-1-1-0-released/ When Elasticsearch version 1.0.0 was released, it came with a new feature: > Snapshot & Restore. The Snapshot portion of this feature allows you to > create backups by taking a “picture” of your indices at a particular

Re: Changing Kibana-int based on context

I don't think you can do this dynamically within kibana. The better way would be to run multiple instances of KB and then use a proxy to handle the redirects. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 14 June 2

better places to store es.nodes and es.port in ES Hive integration?

Hi, I am playing with elasticsearch and hive integration. The documentation says to set configuration like es.nodes, es.port in TBLPROPERTIES. It works. But it can cause many reduntant codes. If I have ten data set to index to the same es cluster, I would have to repeat this information ten

Re: RepositoryMissingException

Well, that was it. I copied the sample PUT from the elasticsearch web site, which of course uses curl, and did not take out the -d. Definitely helps to have another pair of eyes. I was looking at that all day and didn't see the -d. Thanks for your help. Shawn On Friday, June 13, 2014 5:35

Re: Showing stats from delete operation

You will need to raise a github request for this. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 14 June 2014 08:41, wrote: > Would it be possible to add some stats to the response from a > DeleteByQuery giving in

Re: Linear Scaling with ES

The answer is - it depends. If you can provide a bit more detail on what you've done, your setup etc, maybe someone can provide more assistance. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 14 June 2014 07:48, pr

Showing stats from delete operation

Would it be possible to add some stats to the response from a DeleteByQuery giving information on how my objects were deleted? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, se

index.cache.filter.type

I'm toying with the effects of different settings and noticed that setting `index.cache.filter.type: none` works fine, but setting `index.cache.filter.type: soft` or `index.cache.filter.type: weak` gives me stack traces. Am I doing it wrong? The docs mention soft, weak and resident being the type's

Linear Scaling with ES

Hi, We have been spending considerable amount of time now just to figure out if we can get linear scaling in ES by increasing number of nodes or shards or some other parameters. We did so many experiments, changing shards, changing nodes, changing replica, etc but looks to me with everything we

Re: Securing Data in Elasticsearch

You should start HTTP only on localhost then and run Kibana on a selected number of nodes only. There are some authentication solutions for Kibana. I am not able to find security features like audit trails or preventing writes in Kibana/ES so you have to take care. Assessing Kibana for attacks ov

HIVE-Elasticsearch [mapr-elasticsearch] write to elasticsearch issue

Hi , I am trying to integrate elasticsearch with a mapr hadoop cluster. I am using the hive-elasticsearch integration document. I am able to read data from the elasticsearch node. However I am not able to write data into the elasticsearch node which is my primary requirement. Request to kindly

Re: ingest performance degrades sharply along with the documents having more fileds

Hi, Mark: We are doing single document ingestion. We did a performance comparison between Solr and Elastic Search (ES). The performance for ES degrades dramatically when we increase the metadata fields where Solr performance remains the same. The performance is done in very small data set (ie.

Re: issue of elasticsearch-hadoop-2.0.0 with Hive (cloudera and hortonworks), helps are needed

Thank you for the response. The localhost and 192.168.128.1 are the actually the same ES host. I installed ES cloudera vm on xp. I will try your suggestion though and report back. I will try the table without timestamp column. Sent from my iPhone > On Jun 13, 2014, at 1:59 PM, Costin Leau wrot

elasticsearch-php auth error

Hi, I've been using the php client successfully with a remote server, and I've set up a new server and run into auth problems using the PHP client library. $clientParams['connectionParams']['auth'] = array( 'user', 'pw', 'Basic' ); My issue is now I get back a 401 Authentication Required every

Re: Securing Data in Elasticsearch

ES nodes would be locked down and accessible only to authorized users on the OS level; it's the ability to delete and update indices/documents remotely that's worrisome in this case. Disabling HTTP REST API completely is not possible since it's required by Kibana (running behind a reverse prox

Re: Kibana 3 and changing the default field from _all to message

Ok, it's not a Kibana issue, but my Elasticsearch configuration issue. I could fix it in the elasticsearch.yml file, but I believe it's much safer to fix it in my less-likely-to-be-altered start-up script wrapper. So now when I start ES via the bin/elasticsearch script, but only on behalf of th

Re: issue of elasticsearch-hadoop-2.0.0 with Hive (cloudera and hortonworks), helps are needed

Hi, Sorry for the delayed response, travel and other things got in the way. I have tried replicating the issue on my end and couldn't; see below: On 6/8/14 8:03 PM, elitem way wrote: I am learning the elasticsearch-hadoop. I have a few issues that I do not understand. I am using ES 1.12 on W

Re: exclude some documents (and category filter combination) for some queries

Currently not possible. Elasticsearch will return all the nested documents as long as one of the nested documents satisfies the query. https://github.com/elasticsearch/elasticsearch/issues/3022 The issue is my personal #1 feature requested. Frustrating considering there has been a working impleme

Re: Cassandra with JDBC river plugin

Ok. Thanks it seems I have to make the Cassandra river work. On 13 Jun 2014 16:34, "joergpra...@gmail.com" wrote: > The Cassandra Java Driver is not a JDBC driver. > > Jörg > > > On Fri, Jun 13, 2014 at 11:11 AM, Abhishek Mukherjee <4271...@gmail.com> > wrote: > >> Checking the Elasticsearch log

Re: Configuring YML files Location

For example, I keep my Elasticsearch configurations for use with the ELK stack within this directory: */opt/config/elk/current* So my start-up script calls the elasticsearch command as follows: $ES_HOME/elasticsearch -d ... -Des.path.conf=*/opt/config/elk/current* ... Hope this helps! Brian

Re: Non-Uniform Drive Space Across Nodes

OP here. My numbers on the disk space were not an actual observation of current sizes. It was more of a hypothetical of what can I expect ES to do if I only had three servers and that was the starting disk space available in each. -- You received this message because you are subscribed to t

Kibana 3 and changing the default field from _all to message

I have this typical document being indexed by logstash. The following shows the document in rubydebug mode and not as JSON, but when converted to JSON and indexed the field names and values are the same (in other words, the syntax below isn't one-line JSON but it's clearer to read): { "m

Percollation limits

Hi I wanted to ask those who use percollation: how many queries are you percollating? I need to set up some equivalent of percollation for about 100k queries. With some filtering probably up to 10k would actually had to be checked for each new document. Is the idea of using ES percollations for

Changing Kibana-int based on context

I am a newbie to Computer science in general and at present I am working on a project which involves Elasticsearch, logstash, and Kibana and we are using this to build up a centralized Logging system. In kibana config.js , there is a parameter kibana_index whose default value is set to "Kiban

Re: Cannot Increase Write TPS in Elasticsearch by adding more nodes

I haven't seen it asked yet; what is feeding data into your elasticsearch? Depending on what you're doing to get it there, a large document size could easily bottleneck some feeding mechanisms. It's also noteable that some "green" spinning disks top out in the realm of 72MB/s. It might be useful

Re: Marvel 1.2.0 java.lang.IllegalStateException

Ok works thanks On Friday, 13 June 2014 10:02:06 UTC-4, Paweł Krzaczkowski wrote: > > Hi, > > Yes it's been released as Marvel 1.2.1 > > > 2014-06-13 16:01 GMT+02:00 John Smith >: > >> Is the released? Or it's still in github? >> >> Experiencing the same thing... >> >> Ran the commands from above.

Re: Marvel 1.2.0 java.lang.IllegalStateException

Hi, Yes it's been released as Marvel 1.2.1 2014-06-13 16:01 GMT+02:00 John Smith : > Is the released? Or it's still in github? > > Experiencing the same thing... > > Ran the commands from above... > > http://pastebin.com/WUTTLgsS > > > On Monday, 9 June 2014 14:44:17 UTC-4, Paweł Krzaczkowski w

Re: Marvel 1.2.0 java.lang.IllegalStateException

Is the released? Or it's still in github? Experiencing the same thing... Ran the commands from above... http://pastebin.com/WUTTLgsS On Monday, 9 June 2014 14:44:17 UTC-4, Paweł Krzaczkowski wrote: > > It works .. thx for a quick fix > > > 2014-06-09 17:48 GMT+02:00 Paweł Krzaczkowski >: > >>

No Node Available

Hi guys, I googled about NoNodeAvailableException, but none answers for my questions until now. I'm getting this error when the ES connections between Server and Client are idle during a long time. I saw the number of connections in 9300 port and there is a huge opened sockets number (something

Re: compresstion in ES 1.2.1

Hello Jorg, I am sorry, there was some problem in the implementation at my end. Thanks a lot guys for the insight and help. Appreciate the quick responses. Thanks and Regards Sri On Sunday, June 8, 2014 5:04:24 PM UTC-4, sri wrote: > > Hello Jorg, > > Thanks a lot for the info., i tried applyin

Configuring YML files Location

Hi, I am trying to setup the configuration of ES (elasticsearch.yml and logging.yml) outside of the elasticsearch package. I have put the two files in a separate location and pointed the CONF_DIR to that location. I launch the ES server by specifying the cluster name and node name. The problem I

Re: Runtime JRE?

Yes, you can use Java Server JRE. It is a build without Java desktop graphics library (aka headless JVM). Jörg On Fri, Jun 13, 2014 at 1:53 PM, wrote: > I know the guide says the following: > > While a JRE can be used for the Elasticsearch service, due to its use of a > client VM (as oppose to

Re: Query multiple strings in a field in kibana3?

So no way to store the query itself? I will have save the entire dashboard? On Fri, Jun 13, 2014 at 4:35 PM, Mark Walkom wrote: > You can save dashboards with the query, if that is what you want. You will > need to save one per query though. > > Regards, > Mark Walkom > > Infrastructure Enginee

Runtime JRE?

I know the guide says the following: While a JRE can be used for the Elasticsearch service, due to its use of a client VM (as oppose to a server JVM which offers better performance for long-running applications) its usage is discouraged and a warning will be issued. But I noticed something on

Email alert after threshold crossed logstash?

I am using logstash, elasticsearch and kibana to analyse my logs. I am alerting via email when a particular string comes into the log via email output in logstash: email { match => [ "Session Detected", "logline,*Session closed*" ] ... } This works fine. Now

Re: RepositoryMissingException

good question. that is what is being returned when I make the call. but your question gave me an idea as to what the problem is. thanks. On Jun 12, 2014 11:32 PM, "David Pilato" wrote: > What is this -d in "statlogs -d"? > > -- > David ;-) > Twitter : @dadoonet / @elasticsearchfr / @scrutmydoc

Re: Query multiple strings in a field in kibana3?

You can save dashboards with the query, if that is what you want. You will need to save one per query though. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 13 June 2014 18:15, Siddharth Trikha wrote: > I am usin

Re: Cassandra with JDBC river plugin

The Cassandra Java Driver is not a JDBC driver. Jörg On Fri, Jun 13, 2014 at 11:11 AM, Abhishek Mukherjee <4271...@gmail.com> wrote: > Checking the Elasticsearch log files I found this. > > No suitable driver found for jdbc:cassandra:// > 192.168.1.103:9160/transactionlogdb > at java.sql.Driver

Re: How ElasticSearch nodes syncrhonise in Cluster when nodes have different Index mappings

That depends on how you do the migration, it's not something ES handles automatically, you need to do it yourself. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 13 June 2014 19:47, Bhupali Kalmegh wrote: > Yes, I

How we can use different analyzers for one field.(At a time only one analyzer we will use in search but, search requirement is differ)

Hi, I have below Requirement. Please help me. I am using *elasticsearch-1.1.0* - In Index, I have n no.of Fields. and m no.of Types - Eg: Types: Person,Book - Eg: Fields: - Person: Name,age,Email,Phone - Book: Name,author,price - How to set the analyzers to all Fields and all t

Re: ES 1.2.1 sort by _timestamp

This is just to debug this, to make sure results are indeed not sorted by _timestamp, as you claim. Probably easier to just set _timestamp to stored. -- Itamar Syn-Hershko http://code972.com | @synhershko Freelance Developer & Consultant Author of RavenDB in Actio

Re: ES 1.2.1 sort by _timestamp

On Friday, June 13, 2014 10:31:53 AM UTC+2, Itamar Syn-Hershko wrote: > > Possibly, because it's not provided in the _source, or just use this: > http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-timestamp-field.html#_path_2 > > So your suggestion is to have my app fi

Re: How ElasticSearch nodes syncrhonise in Cluster when nodes have different Index mappings

Yes, I am creating new index and then migrating the data from older index to new index. So, when this migration is going on, if any request comes, then what would be the behaviour? On Friday, June 13, 2014 3:11:52 PM UTC+5:30, Luis García Acosta wrote: > > Mapping is applied at cluster level, a

How ElasticSearch nodes syncrhonise in Cluster when nodes have different Index mappings

Mapping is applied at cluster level, and existing index wont get the new mapping. You will need to reindex your data, aka create a new index after you apply the new mapping -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from t

Re: does document database means denormalize

Yes, definitely think in terms of denormalizing. Joins are hard/expensive in elasticsearch so you need to avoid needing to joing by prejoining. But you have other options as well, see http://www.elasticsearch.org/blog/managing-relations-inside-elasticsearch/ So, say you had a person table and

Re: Need help, multiple aggregations with filters extremely slow, where to look for optimizations?

come back with results > > curl -XPOST " > http://10.129.2.42:9200/logs-idx.20140613/event/_search?search_type=count"; > -d' > { > "query": { > "filtered": { > > "filter": { > "or": [ >

How ElasticSearch nodes syncrhonise in Cluster when nodes have different Index mappings

Hi , Kindly help me to understand the behaviour of ES nodes in Cluster when nodes have different Index mappings. I have 2 ES nodes both are currently having same Index versions. Now I want to upgrade both the nodes with the new index mapping. Scenario 1 : Without keeping the node down, start m

Re: Cassandra with JDBC river plugin

Checking the Elasticsearch log files I found this. No suitable driver found for jdbc:cassandra://192.168.1.103:9160/transactionlogdb at java.sql.DriverManager.getConnection(DriverManager.java:689) at java.sql.DriverManager.getConnection(DriverManager.java:247) at org.xbib.elasticsearch.river.jdb

Mapping for a hash map

Hi there, I'd like to define a mapping for a hash map but I do not manage to get it right. Here is the kind of documents I'd like to index: { "message" : "Elasticsearch test 1", "dates": { "create": "2014-01-11", "update": "2014-06-12" } } { "message" : "Elasticse

Re: Need help, multiple aggregations with filters extremely slow, where to look for optimizations?

So I restructured my curl as follows, is this what you mean?, by doing some first hits i do get some slight improvement, but need to check into production data: Thank you will try it and come back with results curl -XPOST "http://10.129.2.42:9200/logs-idx.20140613/event/_search?search

Re: ES 1.2.1 sort by _timestamp

Possibly, because it's not provided in the _source, or just use this: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-timestamp-field.html#_path_2 -- Itamar Syn-Hershko http://code972.com | @synhershko Freelance Developer & Consultant

Re: ES 1.2.1 sort by _timestamp

On Thursday, June 12, 2014 6:52:16 PM UTC+2, Itamar Syn-Hershko wrote: > > This is weird. Are you sure what you are seeing is not overridden > documents (can happen if you specify the ID yourself)? Can you add the > _timestamp field to the results and verify the documents are indeed not > sort

Query multiple strings in a field in kibana3?

I am using Logstash 1.4.1, elasticsearch 1.1.1, kibana 3.1 for analyzing my logs. I get the parsed fields (from log) in Kibana 3. Now, I have often query on a particular field for many strings. Eg: auth_message is a field and I may have to query for like 20 different strings (all together or

Cassandra with JDBC river plugin

Hi Everyone, I am trying to move data from Cassandra to Elasticsearch. Initially I tried the cassandra-river at https://github.com/eBay/cassandra-river. However I got timed out error which I suspect was originating from the Hector API. I posted a question on ths thread https://groups.google.co

Re: Java API ES 0.90.9 Array (2 elements) in search result gets only one value in SearchHitField.getValues()

I've tested it with ES 1.1 and the described behaviour is gone. So the Java API does a correct interpretation of the JSON search result. On Thursday, January 30, 2014 11:23:04 PM UTC+1, Martin Pape wrote: > > Thanks for the information. I still have some months till production, so > might workar

Re: Need help, multiple aggregations with filters extremely slow, where to look for optimizations?

alhost:9200/logs-idx.20140613/event/_search?search_type=count"; - > d' > { > "aggs": { > "f1": { > "filter": { > "or": [ > { > "and": [ >

Re: Accessing Search Templates via Rest

so i guess its not possible? Am Dienstag, 10. Juni 2014 16:58:31 UTC+2 schrieb Sebastian Gräser: > > Hello, > > maybe someone can help me. Is there a way to get the available search > templates via rest api? havent found a way yet, hope you can help me. > > Best regards > Sebastian > -- You rec

does document database means denormalize

What I am asking is Do different design decisions apply in elasticsearch compared to relational Is denormalized better for elasticsearch -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving ema

Re: Need help, multiple aggregations with filters extremely slow, where to look for optimizations?

Below is an example aggregation i perform, is there any optimizations I can perform? Maybe disabling some features i do not need etc. curl -XPOST "http://localhost:9200/logs-idx.20140613/event/_search?search_type=count"; -d ' { "aggs": { "f1&quo

How To Disable Recovery Process / Delete Old Shards

During a botched upgrade process my data was deleted. As it was a test server it didn't matter. However, upon reinstall it just constantly tries to recover old shards, even after deleting every know file on the server that contains elasticsearch data. Can someone let me know of how to disable the r

Re: Need help, multiple aggregations with filters extremely slow, where to look for optimizations?

Can you show us what your request looks like? (including query and aggs) On Fri, Jun 13, 2014 at 9:09 AM, Thomas wrote: > Hi, > > I'm facing a performance issue with some aggregations I perform, and I > need your help if possible: > > I have to documents, the *request* and the *event*. The requ

Need help, multiple aggregations with filters extremely slow, where to look for optimizations?

Hi, I'm facing a performance issue with some aggregations I perform, and I need your help if possible: I have to documents, the *request* and the *event*. The request is the parent of the event. Below is a (sample) mapping "event" : { "dynamic" : "strict", "_parent" : { "type" : "re