Re: Pushing bulk data to ES in a stream

What kind of data, what keys and values, are you fetching? Note, if you have new fields, ES needs time to create the dynamic mapping. Also, if you index into a new index, ES needs time to create the index. Another point is the rendezvous, if you start a client, it need time to connect with all the

Aggregation on boolean

Hey there! I got one index with two types. I want to do a aggragtion query. This is my query. GET index1/type1,type2/_search { "query": { "match_all": {} }, "size" : 0, "aggs": { "myaggregation": { "terms": { "field": "boolean_field" }

Re: Aggregation on boolean

Sry! It was caused by different indexes. Am Montag, 11. August 2014 10:18:25 UTC+2 schrieb Fabian Köstring: > > Hey there! > > I got one index with two types. I want to do a aggragtion query. > This is my query. > > > GET index1/type1,type2/_search > { >"query": { > "match_all": {} >

Re: Fields with same name as recognized types

Yes, that are the commands that should reproduce this. If on your version works ok, that my question is why on my setup doing a top on long field returns the error ( - error: ReduceSearchPhaseException[Failed to execute phase [fetch], [reduce] ]; nested: ClassCastException[org.elast

Re: Fields with same name as recognized types

I am not sure why your setup is producing this issue. To help to diagnose the cause could you please post the mappings for the index (see [1] for how to do this) and the complete cURL command for your search request? Thanks Colin [1] http://www.elasticsearch.org/guide/en/elasticsearch/refere

Scripting updates

I am new to elastic searcj I see that elasticsearch has switched to a new scripting language yet the update api is still referencing MVEL(http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-update.html#docs-update). Is this still the correct updating mechanism to use? A

Re: Scripting updates

Even after adding "script.disable_dynamic: true" to the config and restarting my node I get the same error. On Monday, August 11, 2014 11:30:12 AM UTC+2, Piro Vorster wrote: > > I am new to elastic searcj > > I see that elasticsearch has switched to a new scripting language yet the > update ap

Re: Scripting updates

After adding script.disable_dynamic: false to my settings it now works. I would still however want to know if I should be using mvel for updating or if there is some other updating mechanism. On Monday, August 11, 2014 11:30:12 AM UTC+2, Piro Vorster wrote: > > I am new to elastic searcj > > I s

Re: JDK version 1.7.0_65?

Do not use Java 7u65. It breaks Google Guava (used by Elasticsearch) and also Groovy (which is planned for ES 1.4 being the default), and maybe more. See also http://www.takipiblog.com/2014/08/07/oracles-latest-java-8-update-broke-your-tools-how-did-it-happen/ https://bugs.openjdk.java.net/brows

Relating two Types in ES based on common field

Hi, We have two types on different indexes. There is a situation where we need to query two types based on a common field. We are looking for a single query to get the list of documents from both the types based on the common field. Your help will be highly appreciated. 1. Type “location”

Paging for nested objects.

Hello, I'm new in elasticsearch and I have a question. How can I do paging for nested object and is it possible? (I need to get comments with paging) *My mappping:* { "test": { "mappings": { "completion": { "properties": { "suggest": {

Kibana time offset

Hi there ! I have log files with a time such as : 14:02:00 I configured .conf files to see on the console output that the time set in my customized timestamp field (mytimestamp) is : 14:05:00 (timezone=>"+00:00") So what do I have to search in Kibana with a two-hour offset ? Meaning that if I

Re: System Requirements for ElasticSearch stack

You can use any machine you want, bare metal, VM, whatever. ES is not bound to Amazon EC2 conditions at all. It will depend on your data. Jörg On Fri, Aug 8, 2014 at 3:48 AM, vjbangis wrote: > Thanks Gopinath! > > Hi Jörg, > > *For fault tolerance, you should take into consideration the availa

Re: field boosting at query time

Hi, If I'm not wrong you cannot boost fields in _all field on query time. Querying the _all field should take in account the index-time boost, but for query-time boost you will need the field names you are searching in. You can set boost for a field at query time using "^": "title^5.0" You can pr

Why did "some" ElasticSearch Dashboards disappeared after upgrade?

I am using ElasticSearch for indexing logs. I have configured few Kibana dashboards to periodically check the health of the logs. I upgraded the ElasticSearch nodes from "" to "". Some of the dashboards in Kibana have stopped working since the upgrade. The error following error is shown: Error

Re: Why did "some" ElasticSearch Dashboards disappeared after upgrade?

Sorry. Missed the versions. We upgraded from 1.2.0.1 to 1.3.1. On Monday, August 11, 2014 5:20:58 PM UTC+5:30, Sabyasachi Ruj wrote: > > I am using ElasticSearch for indexing logs. I have configured few Kibana > dashboards to periodically check the health of the logs. > > I upgraded the ElasticSe

Re: Pushing bulk data to ES in a stream

Thanks a lot. I understand the few bottlenecks necessitated while a client is initialized. We have a tight schema for our docs wherein all our docs follow a strict schema already enforced before indexing starts to take place. You can image a sample of our doc to have 32 odd keys with 20 of the

[ANN] elasticsearch-zookeeper for elasticsearch 1.3

Hi, I've just released my changes to the elasticsearch-zookeeper plugin for compatibility with elasticsearch 1.3. You may find it at https://github.com/grmblfrz/elasticsearch-zookeeper/releases. Maybe someone will find it useful. Greetings, Swen -- You received this message because you are su

Absolute scoring of fields

Hey there! I have some trouble with building a query in elasticsearch.. My requirements are: Documents have 4 fields: Field_A Field_B Field_C Field_D I would like to build an absolut scoring, that DOESN'T take the elasticsearch scoring into account. If the search query matches one field, the

Re: [ANN] elasticsearch-zookeeper for elasticsearch 1.3

Hello Swen , I am just curious , what are the proven advantages of using this plugin ? Thanks Vineeth On Mon, Aug 11, 2014 at 6:20 PM, Swen Thümmler wrote: > Hi, > > I've just released my changes to the elasticsearch-zookeeper plugin for > compatibility with elasticsearch 1.3. You

Re: [ANN] elasticsearch-zookeeper for elasticsearch 1.3

Am Montag, 11. August 2014 15:04:52 UTC+2 schrieb vineeth mohan: > > Hello Swen , > > I am just curious , what are the proven advantages of using this plugin ? > Hello Vineeth, in my experience the plugin is much more reliable than zen-discovery. With zen-discovery I was frequently confronted

Re: Pushing bulk data to ES in a stream

Websockets are raw TCP/IP socket. If you want a comparison with bulk indexing over HTTP (port 9200), you will find that if you set HTTP keep-alive you will get similar performance patterns, and with the transport protocol (port 9300), you can set TCP socket keepalive and port reuse flags, which ens

clarity for shard allocation disable/enable during upgrade

I have 8 data nodes and 6 coordinator nodes in an active cluster running 1.2.1 I want to upgrade to 1.3.1 When reading http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html the upgrade docs am I correct to assume: a) disable shard allocation before doing a

Weird Exception.

Hi using ElasticSearch 1.3.0 JAVA 1.8_5 configed for 32g [2014-08-11 09:23:16,258][WARN ][cluster.action.shard ] [Tyrant] [.marvel-2014.08.04][0] received shard failed for [.marvel-2014.08.04][0], node[TsoETYSERg-DNDpiDxpxKA], [R], s[INITIALIZING], indexUUID [gxfk0pCiQg2QCJRyUWAYTw], reason

Re: Weird Exception.

And I see this also... [2014-08-11 09:31:35,233][WARN ][cluster.action.shard ] [Scarlet Spiders] [.marvel-2014.08.04][0] sending failed shard for [.marvel-2014.08.04][0], node[TsoETYSERg-DNDpiDxpxKA], [R], s[INITIALIZING], indexUUID [gxfk0pCiQg2QCJRyUWAYTw], reason [engine failure, message

Re: Weird Exception.

Oops switched linux boxes and forgot to set the file limit. Let me see if that works :) On Monday, 11 August 2014 09:33:54 UTC-4, John Smith wrote: > > And I see this also... > > [2014-08-11 09:31:35,233][WARN ][cluster.action.shard ] [Scarlet > Spiders] [.marvel-2014.08.04][0] sending faile

Re: Weird Exception.

Ok set my sysctl and user limits to 65536 On Monday, 11 August 2014 09:37:33 UTC-4, John Smith wrote: > > Oops switched linux boxes and forgot to set the file limit. Let me see if > that works :) > > On Monday, 11 August 2014 09:33:54 UTC-4, John Smith wrote: >> >> And I see this also... >> >> [2

Re: Why did "some" ElasticSearch Dashboards disappeared after upgrade?

I just tried this: http://www.elasticsearch.org/blog/tool-help-routing-issues-elasticsearch-1-2-0/ This tool fails with the following error: $ java -jar elasticsearch-fix-routing-1.0.jar localhost 9300 grafana copy_if_missing Aug 11, 2014 1:46:38 PM org.elasticsearch.plugins INFO: [Donald Pier

Re: Weird Exception.

Ok set my sysctl and user limits to 65536. Node restarted seems to be recovering so far... On Monday, 11 August 2014 09:37:33 UTC-4, John Smith wrote: > > Oops switched linux boxes and forgot to set the file limit. Let me see if > that works :) > > On Monday, 11 August 2014 09:33:54 UTC-4, John

Elasticsearch inserting date type documents as UTC timezone datetime while indexing

Hi, I'm using below code to insert some documents in elasticsearch index. But when inserting in the es the time is coming as UTC format rather than original GMT+5:30Z format. Where as the Sysout before indexing is giving me correct format as *2014-08-11T18:23:13.447+05:30* . Please let me know

Bulk - insert if not exists, update otherwise?

I have a situation where I have code that does 3000 bulk inserts of data. This works just fine. What I now need to do is 3000 bulk inserts OR updates. Specifically, if the key already exists, I need to UPDATE fields A, B and C. If the key does NOT exists, I need to INSERT all fields. But again,

Usage Reports in ElasticSearch?

I haven't seen anything that captures users activity and reporting. I am interested in users events, logging, what searches, date od search, most popular searches, products most searched...etc. Is there anything in Elastic search/plugins that helps me do this? Or, should I be tracking all this

[ANNOUNCE] kibana-rack 0.1.3 released (another important bugfix)

Hello all, Unfortunately there was a regression in 0.1.1 where path that static dashboard files were served from was incorrect. If you downloaded version 0.1.1 of kibana-rack after the previous post, please upgrade to 0.1.3 (https://github.com/tabolario/kibana-rack/releases/tag/v0.1.3) immediat

Re: Embedded ElasticSearch On Java

All, I know this post is old, but I continue to have an issue with this... I get an NoSuchMethodError: org.apache.log4j.Logger.isTraceEnabled()Z exception when I run Node node = NodeBuilder.nodeBuilder().local(true).node(); // exception thrown here... Client client = node.client(); I have tr

Re: [Shard Unassigned] Solutions to reassigned the shard

Yeah, I agree. It is kind of frustrating, as I know it is on the right place but ES gives me still YELLOW ! I wish It was only a matter of modifying a single file in the index directory. Can someone share its experience on the matter ? Thanks, Alex. 2014-08-09 2:23 GMT+02:00 Mark Walkom : > I

Re: Embedded ElasticSearch On Java

Have you tried using transport client for connecting... On Monday, August 11, 2014 10:26:29 PM UTC+5:30, Kfeenz wrote: > > All, > > I know this post is old, but I continue to have an issue with this... > > I get an NoSuchMethodError: org.apache.log4j.Logger.isTraceEnabled()Z > exception when I

Bulk - insert if not exists, update otherwise?

What you're describing is the upsert functionality in the mvel scripting. The upsert will create and populate when the key doesn't exist. And the update api will add to the document if it does already exist. -- You received this message because you are subscribed to the Google Groups "el

using a nested object field in a multi_match query

Hello, I'm having trouble coming up with how to supply a field within a nested object in the multi_match fields list. I'm using the multi_match query in order to perform query time field boosting, but something like: "query": { "multi_match": { "query": "China Mieville", "oper

Re: Bulk - insert if not exists, update otherwise?

Excellent - are there any examples I can see? Where can I read and learn how to do this? On Monday, August 11, 2014 10:19:51 AM UTC-7, eune...@gmail.com wrote: > > What you're describing is the upsert functionality in the mvel scripting. > > The upsert will create and populate when the key does

Re: Bulk - insert if not exists, update otherwise?

Wait, hang on - I just saw this in the docs. Are you suggesting a solution that's being deprecated? If so, that's likely not a good idea. I'm now confused ;) [image: Warning] Deprecated in 1.3.0. Mvel has been deprecated and will be removed in 1.4.0. -- You received this message because you a

Failed to perform snapshot (index files)]; nested: FileNotFoundException

Hello I have a 1.2.2 cluster of 6 nodes with several indexes configured with 2 to 4 replicas. I'm trying to perform a snapshot but it completes only partially with failures like this: CreateSnapshotResponse[snapshotInfo=SnapshotInfo[name=2014-08-11-16-31-04, state=PARTIAL,reason=,indices=Object[

Re: Embedded ElasticSearch On Java

So I am very new to elasticsearch... so I apologize in advance.. I started a local instance of elasticsearch and I am trying to connect to it through the Java API. I was under the impression that the transport client was for remote clients? I tried: @Test public void testIndexResponse() { C

Re: possible problem in yml config file

indentation is okay so that' not it. On Fri, Aug 8, 2014 at 7:54 PM, David Pilato wrote: > Check indentation. YAML is very sensible. > > So if you have: > > cluster.name: bla > http.max_content_length: 100mb > > The later won't be applied. It must be: > cluster.name: bla > http.max_content_le

Role of dedicated master nodes when not elected

Lots of ES best practice articles recommend having dedicated master nodes. Specifically, that would involve setting these flags: node.master: true node.data: false Say, you had 7 index nodes and 3 master nodes (https://blog.hipchat.com/2013/10/16/how-hipchat-scales-to-1-billion-messages/) in

Re: Role of dedicated master nodes when not elected

As a followup, any ratio guidelines for indexing nodes vs dedicated masters. From what I can tell, it wouldn't make much sense to run with only one-dedicated master node; if that node goes down, your whole cluster becomes unavailable. On Monday, August 11, 2014 2:18:08 PM UTC-4, Matt Hughes

ES service self update?

I am very new to Elastic Search. I was curious if ES service can periodically check for the latest version such as 1.1.3 and update nodes one at a time silently? I am in the process of writing PowerShell script to do this but thought ES might have solution for this already. thanks -- You rece

Re: Pushing bulk data to ES in a stream

Thanks a lot for the response. On Saturday, August 9, 2014 1:15:17 AM UTC+5:30, abhiji...@housing.com wrote: > > Hello everyone, > > I wanted to know if it is possible to index the docs through a stream > which pushes data to the Elasticsearch cluster. > > Our current problem is to index the hu

ENOMEM error on ES

Hi all, I just moved to start using elasticsearch as a non-root user. I am getting this error at startup *[2014-08-08 22:38:17,768][WARN ][common.jna ] Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out. Increase RLIMIT_MEMLOCK (limit).*

Re: Bulk - insert if not exists, update otherwise?

I see what you mean about MVEL I guess the same functionality is available in the replacement. Groovy. I will need to investigate switching to Groovy. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop

Ability to filter the results with a configurable amount of lines around an event

one big aspect of log analysis is to pinpoint a problematic event and than see the surrounding events. Is there a way to query for X number of event before and after the event ? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe fr

Re: Embedded ElasticSearch On Java

1. Never use constructs like " ... = new TransportClient()..." This uses a new client per call which is very inefficient. Use a singleton per JVM instead. Do not forget to close the TransportClient thread pool when JVM shuts down with client.threadPool().close() 2. Use TransportClient settings, yo

Re: Ability to filter the results with a configurable amount of lines around an event

You can do this using the timestamp (with a range, and grow it if necessary) or if you have a serial ID of some sort on the log message you can do a range query on that -- Itamar Syn-Hershko http://code972.com | @synhershko Freelance Developer & Consultant Author

Re: Bulk - insert if not exists, update otherwise?

Okay, so now here's where I am - I read up on upsert and crafted my bulk stack. It seems to work: {"update":{"_index":"aftermarket-2014-08-11_02-38-19","_type":"premium","_id":"kryptonblue.com"}} {"script" : "ctx._source.auctionid=6623102; ctx._source.auctiontype=18; ctx._source.auctionstatus=4;

Strange problem with index delete

This is a strange problem, I am not sure I ended up here. I am on a cluster of 2 nodes, and was trying to delete the index, I keep getting acknowleged = false. If I look at my index through HEAD plugin, I get "index name" size: unknown docs: unknown and I see no shards assigned to either node

org.elasticsearch.search.aggregations docs

I've been looking all over the place for documentation on org.elasticsearch.search.aggregations._ The Java API 1.x docs state Facets will be depreciated. I am using the source code on github for reference. I also see that Kibana does all its queries using Facets. I wanted to make sure I wasn'

Re: Restoring a snapshot from one machine to a different machine.

Still the same issue here as well... Settings are the same on both servers, but after copying (rsync) the files to the other server, the other server fails to see any snapshots in the API call to list them. Same problem as Donald: the respository has only the "indices" sub directory, and the PUT

Re: Bulk - insert if not exists, update otherwise?

I think you're on the right track. If you just run again you should get the update. Does the document appear correct? Note new option in 1.4 scripted_upsert true Allows the document to be sent once for efficiency. -- You received this message because you are subscribed to the Google Groups

Re: ENOMEM error on ES

This is a warning, not an error. As you can see, you allowed only 65536 bytes(!) for memory locking. You can set this to unlimited if you are on Linux OS with memory overcommit. Jörg On Mon, Aug 11, 2014 at 8:50 PM, allen chan wrote: > Hi all, > > I just moved to start using elasticsearch as

Can't delete Index!

I got myself in trouble by changing the mapping. I am on a cluster of 2 nodes, and trying to delete the index, I keep getting acknowleged = false with normal delete curl -X GET "http://localhost:9200/myIndex If I look at my index through HEAD plugin, I get "myIndex" size: unknown docs: unknown

Re: Can't delete Index!

Your JVMs (at least one of them) are not able to process the command, so you should check out the data directory where the file of the index to be deleted are located. Shut your cluster down, remove the index files of the index to be deleted from the filesystem, and start the cluster up again. To

Re: Restoring a snapshot from one machine to a different machine.

The permissions are wide open for the snapshot repository. I followed the directions from the snapshot documentation; I wonder if I'm missing a trailing slash or something, there are various files under different indices underneath the sub directories beneath my /snapshots/indices/{indexname}/

Re: Can't delete Index!

I wish it was that simple, I deleted the directory of the index from /data. Yet, the index is still there and won't be deleted or recreated.. I still get acknowledged = false. On Monday, August 11, 2014 5:31:36 PM UTC-4, Jörg Prante wrote: > > Your JVMs (at least one of them) are not able to pro

Re: Can't delete Index!

I also deleted the directory, restarted the nodes, and it recovered the index still, which is null. On Monday, August 11, 2014 5:42:53 PM UTC-4, Sam2014 wrote: > > I wish it was that simple, I deleted the directory of the index from > /data. Yet, the index is still there and won't be deleted or

Re: Can't delete Index!

Oops, When I restarted the cluster/nodes, I see shards under an "unassigned node", neither of the two nodes of the cluster. Is there a way I can route/assign /delete out of this vicious cycle? On Monday, August 11, 2014 5:46:25 PM UTC-4, Sam2014 wrote: > > I also deleted the directory, restarted

Re: Absolute scoring of fields

You can wrap each individual match query in a constant score query and place them as clauses in a boolean query. The guide has an example: http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/ignoring-tfidf.html#constant-score-query Cheers, Ivan Bernhardt Scherer wrote: >Heyth

Re: Can't delete Index!

There is no vicious circle. First you must stop the cluster. Then, after processes have exited, continue with file operations. After that, you can start the cluster nodes again. Jörg On Mon, Aug 11, 2014 at 11:49 PM, Sam2014 wrote: > Oops, When I restarted the cluster/nodes, I see shards under

Creating filters per aggregation similar to Facets

Kibana provides a good example of date histograms, split out by each "query" entered at the top in the "Query" bar. It essentially creates multiple free text queries against "all". I see it generates per facet filter, with a free text (query_string) search. Since facets are to be depreciated

impact of stored fields on performance

I recently added a binary type field to all documents with mapping "store": "true". The field contents are large and as a result the on-disk index size rose by 3x, from 2.5Gb/shard to ~8Gb/shard. After this change I've seen a big jump in query latency. Searches which previously took 40-60ms

Re: ES service self update?

No it doesn't. You could do it with puppet/chef. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 12 August 2014 04:39, wrote: > I am very new to Elastic Search. I was curious if ES service can > periodically check

Re: Role of dedicated master nodes when not elected

They can act as search nodes, otherwise they don't do much. There are no guidelines other than using an odd number of masters, and as you pointed out, more than just 1. We currently have 3 masters for 25 data nodes. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campa

Re: Usage Reports in ElasticSearch?

There's nothing within ES to do this, you'd have to put a proxy in front that captures it, then build the logic around it. This has been asked before so it'd be worth checking the list archives. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: w

Re: Can't delete Index!

I stopped node 1, then I stopped node 2. I disabled the cluster, deleted files from nodes, Made sure no Indices on either node. I re-enabled cluster in yml, restarted cluster, and the index came back! On Monday, August 11, 2014 6:16:42 PM UTC-4, Jörg Prante wrote: > > There is no vicious circle.

Re: org.elasticsearch.search.aggregations docs

On Mon, Aug 11, 2014 at 10:47 PM, Jeff Steinmetz < jeffrey.steinm...@gmail.com> wrote: > I've been looking all over the place for documentation on > > org.elasticsearch.search.aggregations._ > There's quite a bit of information in the online docs: http://www.elasticsearch.org/guide/en/elasticsea

Re: Can't delete Index!

I stopped node 1, then I stopped node 2. I disabled the cluster, deleted files from nodes, Made sure no Indices on either node. I re-enabled cluster in yml, restarted cluster, and the elusive index came back. I don't know what I am missing! On Monday, August 11, 2014 6:16:42 PM UTC-4, Jörg Pran

Re: Can't delete Index!

Can you post your detailed steps and debug logs? Make also sure there is no third node - this looks like ES is shipping shards to a third node from which the two nodes are recovering from. Jörg On Tue, Aug 12, 2014 at 1:06 AM, Sam2014 wrote: > I stopped node 1, then I stopped node 2. I disabl

Re: org.elasticsearch.search.aggregations docs

Thank you. Although I was specifically talking about documentation for the Java search API. For example, there is this http://www.elasticsearch.org/guide/en/elasticsearch/client/java-api/current/java-facets.html But ... haven't found anything that covers the Aggregations replacement. On Mond

Problem Plotting Zero values with Kibana

Hi, I am using a ELK stack for log processing. One of the problem I am facing with kibana is that, I have some fields in my log whose values is zero, and zero is considered as legitimate value in my case. But when I plot those fields in kibana, if zero field is enabled it plots zero in the grap

Re: Problem Plotting Zero values with Kibana

This was just answered on your post to the LS list, but you can set zero fill under panel settings. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 12 August 2014 09:50, shriyansh jain wrote: > Hi, > > I am using a

Re: Problem Plotting Zero values with Kibana

Hi, Thank you for your reply back. I am aware of the zero fill settings in kibana. But my problem is I have some log values which are zero at particular timestamp and don't have any values at some timestamp. So if I select zero fill it plots zero for both the actual zero values and the values wh

Re: clarity for shard allocation disable/enable during upgrade

When you disable reallocation and then reboot a node, it should simply re-enable the shards locally rather than trying to do so on other nodes. However as you have noticed this doesn't actually occur and I have raised a github issue to get this investigated, so feel free to comment on it as well -

nested fields and _field_names

I am trying to profile fields by counting documents with each given field. I am using 1.3.0 ES, so I thought I can use: aggs: { docs : { terms : { field : _field_names, size : 0 } } } to give a count of documents with each specific field/path. But this returns

Dynamically add GROK rules to logstash

Hi , I have various GROK rules which will identify mark various log lines based on certain rules. Currently on adding a new GROK rule , we need to restart logstash. Is there a way to do it , without restarting logstash , i.e. dynamically change or add the GROK conf file. Thanks Vineeth

Re: Dynamically add GROK rules to logstash

Nope, you need to restart logstash for it to pick up the new config. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 12 August 2014 13:52, vineeth mohan wrote: > Hi , > > I have various GROK rules which will identi

Re: Elasticsearch inserting date type documents as UTC timezone datetime while indexing

Hi, Any ideas how to prevent the time changing while indexing in es, or to convert in correct format while query ? Thanks, Subhadip -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails fro

Index size on node VS heap size

Hi, Let's say my machine has 40G of RAM, and so I set HEAP_SIZE to 20G, as recommended. And let's say I have a single index on the machine. Rougly, how large can the index be to maintain good performance? Must it be somewhat less than 20G? Thank you. -- You received this message

Re: Index size on node VS heap size

No it can be more, it depends on what sort of queries you are doing and what data structures/types you are indexing. Best bet is to keep throwing data at the index until the server can't take it, then you know the limit. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@

Re: strange highlight result, can anyone explain it?

For people that face the same problems, it's because the analyzers used in highlight is included all the ones of the fields in the query if you don't turn require_field_match to true. FYI. Ivan Ji於 2014年8月5日星期二UTC+8下午8時08分31秒寫道： > > The query command I used is as > > {'multi_match': {'fields':

Compression Mechanism

ElasticSearch uses LZF on stored fields (including _source). The storage requirements will depend on your implementation and the complexity of your data, however, planning for a 1:1 ratio +10% with compression enabled aught to put you on the right path. Otherwise, you'll have to experiment to fi

Unassigned shards after updating # of replica

Hi, I am using ES 0.90.13 with 10 clusters. Java Sun 1.7.53. We find out that the # of replicas on our ES clusters was zero. Hence, we increased the # of replica last week on ES2 to 1. Since then the cluster became yellow <-> green. It runs fine for several hours and then became yellow (with

Re: Compression Mechanism

Elasticsearch uses LZ4, see http://blog.jpountz.net/post/35667727458/stored-fields-compression-in-lucene-4-1 For storage requirements, you need around twice the disk space if you incrementally grow your index, because of additional segment merge space overhead. Jörg On Tue, Aug 12, 2014 at 8:49