Re: History or logging for configuration change

Yep, in the ES log. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 21 August 2014 04:54, kti...@hotmail.com wrote: Hi, Is there some sort of logging that Elastic Search writes whenever say a new index

Re: Best Practice - ES and Kibana on Different Machines?

Exactly. It's ultimately up to your own security requirements. We run separate as we have a web server that runs a few instances of kibana and a few other things, which allows finer grained access controls. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma

Re: Ideal setup for EC2 cluster Config

Yes, but there is nothing stopping you from sending queries to these master nodes if you wish. Client nodes can be useful if you do a lot of, or heavy queries. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 21

Re: How to safely migrate from one mount to another mount in Elasticsearch to store the data

Do you want to copy the existing data in /auto/share to /auto/foo, or start with no data? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 19 August 2014 08:23, shriyansh jain shriyanshaj...@gmail.com wrote: Hi

Re: How to safely migrate from one mount to another mount in Elasticsearch to store the data

If you want no data in /auto/foo then just create the directory, give it the right permissions and then update the config to point to it. It's the same process you did for /auto/share. Do you have replicas set on your indexes? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email

Re: How to safely migrate from one mount to another mount in Elasticsearch to store the data

location it will simply start to copy the data from the other node so that you fulfil your replica requirements. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 19 August 2014 08:58, shriyansh jain shriyanshaj

Re: How to safely migrate from one mount to another mount in Elasticsearch to store the data

Why do you want to do this if you are worried about data loss? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 19 August 2014 11:50, shriyansh jain shriyanshaj...@gmail.com wrote: As you mentioned the node

Re: How to safely migrate from one mount to another mount in Elasticsearch to store the data

problems than it's worth. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 19 August 2014 11:59, shriyansh jain shriyanshaj...@gmail.com wrote: Just to make sure if /auto/share goes down I have data in /auto/foo

Re: How to safely migrate from one mount to another mount in Elasticsearch to store the data

Apart from replica's, that's really outside the scope of what ES provides. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 19 August 2014 12:12, shriyansh jain shriyanshaj...@gmail.com wrote: I got your point sir

Re: A few questions about node types + usage

Master, data and client are really just abstractions of different combinations of node.data and node.master values. A node.master=true, node.data=false can handle both cluster management and queries. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com

Re: Blocking the communication between 2 ElasticSearch severs in 4 nodes cluster leads to split brain

This looks similar to https://github.com/elasticsearch/elasticsearch/issues/2488 Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 18 August 2014 00:55, moti.uman...@gmail.com wrote: We have a 4 node cluster

Re: A few questions about node types + usage

use masters that double as clients and they only have 8GB, our use sounds similar and we don't have issues. I wouldn't bother with 3 client only nodes to start, use them as master and client and then if you find you are hitting memory issues due to queries you can re-evaluate things. Regards, Mark

Re: Elasticssearch cluster installation on windows 2012

It should be ok, but if this is production then you should really have an odd number of nodes, ie 3+. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 14 August 2014 21:04, ozlevka ozle...@gmail.com wrote: Hi All

Re: Elasticsearch cluster on AWS. Article.

Nice and indepth, thanks for posting. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 15 August 2014 03:13, Pavel P pa...@kredito.de wrote: Hi everyone, Below you can find one big article, summing up all my

Re: Elasticssearch cluster installation on windows 2012

No, but there appears to be a lot of discussions around it on various other places on the web. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 15 August 2014 02:17, Andrew Gui yang.gu...@gmail.com wrote: Do you

Re: Newbie Query : Can ES be used for searching spatial data

1 - It sure does, have a search through the docs for geo http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index.html 2 - When you send data to ES it indexes it and is then available for search in near-realtime. 3 - That's what it's built for :) Regards, Mark Walkom

Re: how i change the default port 9200 for plugin head? thanks

You can't change it for head alone, you'd have to change it for the entire instance. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 15 August 2014 10:45, huangshan...@gmail.com wrote: how i change the default

Re: Feature request? Ignore allow_explicit_index when accessing root /_bulk URL

That'd be worth entering in here - https://github.com/elasticsearch/elasticsearch/issues :) Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 13 August 2014 22:37, Иван Кадочников fizmat@gmail.com wrote: Hello

Re: Best URL for load balancer HTTP health check

If you just curl http://IP:9200 you will get a response, if it's not 200 then chances are it's not part of the cluster and something is wrong. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 14 August 2014 07:35

Re: Get Shard Info From Cluster/Nodes/Index

The _cat/shards API will tell you this - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cat-shards.html Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 14 August 2014 03:19, 'Sandeep Ramesh

Re: Moving Index/Shards from One Node to Another

and not running replicas is risky, especially if you are forcing the entire index onto one instance. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 13 August 2014 23:08, 'Sandeep Ramesh Khanzode' via elasticsearch

Re: Automatic partial shutdown of cluster

Something is shutting your node down using the API. You might want to look at http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-shutdown.html#_disable_shutdown Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web

Re: Index size on node VS heap size

No it can be more, it depends on what sort of queries you are doing and what data structures/types you are indexing. Best bet is to keep throwing data at the index until the server can't take it, then you know the limit. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma

Re: Compression Mechanism

Data use will depend on your indexing rate and how big your documents are, which is entirely dependant on your use. However ES uses the lz4 compression algorithm. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 11

Re: ES service self update?

No it doesn't. You could do it with puppet/chef. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 12 August 2014 04:39, kti...@hotmail.com wrote: I am very new to Elastic Search. I was curious if ES service can

Re: Role of dedicated master nodes when not elected

They can act as search nodes, otherwise they don't do much. There are no guidelines other than using an odd number of masters, and as you pointed out, more than just 1. We currently have 3 masters for 25 data nodes. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma

Re: Usage Reports in ElasticSearch?

There's nothing within ES to do this, you'd have to put a proxy in front that captures it, then build the logic around it. This has been asked before so it'd be worth checking the list archives. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web

Re: Problem Plotting Zero values with Kibana

This was just answered on your post to the LS list, but you can set zero fill under panel settings. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 12 August 2014 09:50, shriyansh jain shriyanshaj...@gmail.com wrote

Re: clarity for shard allocation disable/enable during upgrade

- https://github.com/elasticsearch/elasticsearch/issues/7117 Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 11 August 2014 23:31, bitsofinf...@gmail.com wrote: I have 8 data nodes and 6 coordinator nodes

Re: Dynamically add GROK rules to logstash

Nope, you need to restart logstash for it to pick up the new config. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 12 August 2014 13:52, vineeth mohan vm.vineethmo...@gmail.com wrote: Hi , I have various GROK

Re: Strange system load increase

How are you measuring the searches/s metric? ES doesn't run searches within itself, they have to be initiated externally somehow. Also, you should really upgrade :) Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com

Re: searching from the big index - Java heap space exception

How much data do you have in your cluster? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 8 August 2014 05:43, Pavel P pa...@kredito.de wrote: Hi, I have a cluster of 3 machines - each 8 CPU and 30Gb ram

Re: how could I reset the cluster configuration value

It's not fixed, it's been moved to https://github.com/elasticsearch/elasticsearch/issues/6732 Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 8 August 2014 21:31, Pavel P pa...@kredito.de wrote: Hi, Once I've

Re: [Shard Unassigned] Solutions to reassigned the shard

It's probably because the cluster state doesn't know about the index even though it can see it on the filesystem. I don't know if you can recover this, but there might be a way someone can suggest. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web

Re: Cluster vs single node

What sort of data are you searching and what sort of searches are they against it? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 7 August 2014 04:20, Geoff geoffreyo...@gmail.com wrote: Hi, First I

Re: Problem starting elasticsearch 0.90.11

I just tested this and didn't run into the same problem. Have you changed something locally around your environment variables, or in the service defaults? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 8 August

Re: Storing Elasticsearch configuraton and deploying new clusters

There are some settings you can set via the API, but not all. The docs are pretty indepth so check out http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web

Re: Storing Elasticsearch configuraton and deploying new clusters

You can store mappings on the filesystem - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-conf-mappings.html Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 6 August 2014 10:03, Hayden

Re: Kibana response time is too slow, need help identifying why

You could check the slow log or hot threads to see if there is anything. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 5 August 2014 07:42, Tony Chong tonyjch...@gmail.com wrote: Hello, Like many others, I

Re: Marvel History

No, you need something like curator - https://github.com/elasticsearch/curator - to handle it. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 5 August 2014 02:43, Daniel Schonfeld downwindab...@gmail.com wrote

Re: If I have ELK stack running on EC2. How can I make the ES as a cluster?

ES can take disk space into account, 1.3.X does this automatically - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-allocation.html#disk Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web

Re: Shard rebalancing

Shard size will depend entirely on how many shards you've set and how big the index is. Allocation of data to shards happens in a round-robin manner, so balancing isn't needed. What do you mean by shards changing in the background? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor

Re: Design HA ES for 16 TB logs data | Is SAN storage a good idea?

Heavy aggregations = lots of ram Storage, if you can use SSD. The only rule of thumb is get the best possible hardware that you can afford. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 4 August 2014 13:09, John

Re: Is this process is strange?

You should really ask this on the logstash list. However its probably the logstash-web service. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 3 August 2014 01:56, vjbangis jessviray0...@gmail.com wrote

Re: Unable to install plugin

Do you have a /usr/share/elasticsearch/bin/plugin file? If so what happens if you run it with the -h flag? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 1 August 2014 15:57, sa...@systeminsights.com wrote: Hey

Re: index size impact on search performance?

You can create one every hour if you want, daily is good. You may have to deal with that in the code however it makes management a lot easier. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 1 August 2014 14:35

Re: Dealing with spam in this forum

I agree with Ivan and Lukas, though recruiters with ES related jobs should be ok as long as it's not a generic BCC like the ones Ivan deleted. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 2 August 2014 05:05

Re: Elasticsearch throwing exception while starting on Redhat

You should really use service to handle starting it - (sudo) service elasticsearch start This will include the correct configurations, /etc/elasticsearch/elasticsearch.yml and /etc/default/elasticsearch. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma

Re: No route to self

Can you put your config into a gist/pastebin? ES isn't binding to loopback is it? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 2 August 2014 11:51, Matt Hughes hughes.m...@gmail.com wrote: Originally I

Re: ES reports 4 nodes but only 3 severs. One node is used twice

Are you using logstash perhaps? If not then check the number of ES processes on your .204 node as there could be two instances running. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 2 August 2014 12:58, David

Re: ElasticSearch memory usage on centralized log clusters

to drop memory use you can disable bloom filtering. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 31 July 2014 19:16, Tim Stoop tim.st...@gmail.com wrote: Hi all, We've been running an ElasticSearch cluster

Re: ElasticSearch memory usage on centralized log clusters

GCG1 is experimental in that it's not recommended by the ES team as you guessed, even if it is supported within java. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 31 July 2014 21:30, Tim Stoop tim.st...@gmail.com

Re: Recommendations needed for large ELK system design

of nodes will have to take place in an election, which reduces the possibility of split brain. If you set the discovery settings then you are also essentially setting the quorum settings. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web

Re: Cluster making

I don't think you need this - ES handles clustering by itself. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 1 August 2014 11:38, arshpreet singh arsh...@gmail.com wrote: On 1 Aug 2014 06:57, Mark Walkom ma

Re: index size impact on search performance?

If you're using time series data then it makes sense to use time based indexes. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 1 August 2014 12:43, David Pilato da...@pilato.fr wrote: Well. I guess it depends

Re: changing number of shards for new indices

It doesn't change existing indexes only new ones. You can either do the setting change via the API or in the config, if you choose the latter you will need a restart. See http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-create-index.html Regards, Mark Walkom

Re: Remote access through SSH

You need to use SSH directly for it, curl won't work. ssh user@host -i ~/.ssh/id_rsa.pub Assuming you have a public key on the server. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 31 July 2014 08:47, Chia-Eng

Re: Configuration Brain Wobbles

Standard response to this is ES is not built for multi DC clustering, but as long as you are aware you are of that then it's fine. Have you looked at http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-allocation.html ? Regards, Mark Walkom Infrastructure

Re: cluster.routing.allocation.enable behavior (sticky shard allocation not working as expected)

I've seen this as well Ivan, and have also had a few people on IRC comment on the same thing - shards that are local are not simply being initialised, but being reallocated elsewhere. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web

Re: Remote access through SSH

You may want to look at http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search.html If you are just learning ES, then check out http://exploringelasticsearch.com/ Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web

Re: Memory Explosion: Heap Dump in less than one minute

What java version? How much heap have you allocated and how much RAM on the server? Basically you have too much data for the heap size, so increasing it will help. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 31

Re: Memory Explosion: Heap Dump in less than one minute

Up that to 1GB and see if it starts. 512MB is pretty tiny, you're better off starting at 1/2GB if you can. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 31 July 2014 10:28, Tom Wilson twilson...@gmail.com wrote

Re: Memory Explosion: Heap Dump in less than one minute

Unless you are attached to the stats you have in the marvel index for today it might be easier to delete them than try to recover the unavailable shards. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 31 July 2014

Re: Remote access through SSH

You can also curl from your local machine to the server, without having to SSH to it - curl -XGET http://IPADDRESS:9200/ You don't need to provide SSH credentials for that transport client example. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma

Re: Recommendations needed for large ELK system design

very write heavy, with an average of 1K events p/s and comparatively minimal reads. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 31 July 2014 01:35, Alex alex.mon...@gmail.com wrote: Hello, We wish to set up

Re: bulk indexing - optimal refresh_interval

I'd say because if you are inserting a lot of data, you will have a massive hit at the end when you need to index, as opposed to smaller ones along the way. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 29 July

Re: Kibana 3.1.0 vs ElasticSearch 0.90.0 cluster?

Nope - https://github.com/elasticsearch/kibana#requirements Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 30 July 2014 04:08, 'Tian Zhang' via elasticsearch elasticsearch@googlegroups.com wrote: Hi, Kibana

Re: Kibana 3.1.0 vs ElasticSearch 0.90.0 cluster?

You'd have to dig around on the github site and see if there is an older version still available. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 30 July 2014 08:44, 'Tian Zhang' via elasticsearch elasticsearch

Re: 1.1.1 to 1.3 upgrade possible?

Sure is, check out http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html and whatever breaking changes and release notes as applicable. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com

Re: 1.1.1 to 1.3 upgrade possible?

It was just mentioned on IRC there is a bug that may cause problems with mixed version clusters, specifically with 1.3.0, that can cause shards to go unallocated. It's been recommended to wait until 1.3.1 which will contain a fix and is not far off. Regards, Mark Walkom Infrastructure Engineer

Re: If I have ELK stack running on EC2. How can I make the ES as a cluster?

Can you elaborate a bit more, do you only have one node now? It is simple to expand a single node to multiple. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 27 July 2014 19:30, arshpreet singh arsh...@gmail.com

Re: If I have ELK stack running on EC2. How can I make the ES as a cluster?

There's a bit more to it, but yes that is the general idea. I'm not familiar with ES on EC2 though so I can't give you any directions, though something like http://www.elasticsearch.org/tutorials/elasticsearch-on-ec2/ may help. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email

Re: cluster.routing.allocation.awareness.attributes not working as documented

Take a look at the example here http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-cluster.html#allocation-awareness Basically a shard and it's replica will never be allocated to the same instance, which is what you are seeing. Regards, Mark Walkom Infrastructure

Re: cluster.routing.allocation.awareness.attributes not working as documented

If you only have one node the replicas will never be assigned. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 26 July 2014 09:23, Yongtao You yongtao@gmail.com wrote: Hmm... I'm reading the same document

Re: Multimaster architecture

When you define the shards and replica's it's at a cluster level, it doesn't create them on each local server, it spreads them out. Ultimately what you will find is that the number of shards and replicas will be based on which ever nodes is the master at the time. Regards, Mark Walkom

Re: Dealing with spam in this forum

You're better off starting a new thread around this :) (But take a look here http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html ) Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 25

Re: How to safely apply new configuration from elasticsearch.yml

The only way to load the config is with a restart of the service. You could use the cat API to check on the status before reloading - http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/cat-health.html Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma

Re: kibana dashboard save failed

Can you kibana install talk to elasticsearch ok? Are you seeing other data via it? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 23 July 2014 19:52, jin201...@gmail.com wrote: Hello. I just downloaded kibana

Re: Add / Remove nodes in cluster, good practice question

If this is production you really want an odd number of nodes to reduce potential split brain issues. However in your case, just add the new node to the cluster, let it replicate across, then shutdown the node you no longer want. Any impact will be minimal. Regards, Mark Walkom Infrastructure

Re: When to use multiple clusters

for it and it makes things easier to manage. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 24 July 2014 00:34, Alex Kehayias a...@shareablee.com wrote: I have several large indices (100M docs) on the same cluster

ES 1.3.0 and 1.2.3 released

improvements, and bugfixes. We recommend upgrading, especially for users with high indexing or aggregation loads. The full change log is available in the Elasticsearch 1.3.0 release notes http://www.elasticsearch.org/downloads/1-3-0/, but we will highlight the most important changes below: Regards, Mark

Re: How to Cleanly Recover After Crash

It might be easier to just delete the data and reindex if you can (apart from marvel). There are a few lucene tools that you could run, but that's pretty advanced, there isn't anything within ES itself. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma

Re: Solaris 10 mlockall error code

What elasticsearch version are you on? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 21 July 2014 19:09, James Pace james.a.p...@gmail.com wrote: We have been having issues running ES with the bootstrap.mlockall

Re: Handling node failure in ES cluster

Max and min memory should be the same, mlockall is probably not working due to these being different as it can't lock a sliding window. Try setting that and see if it helps. Also you didn't mention your java version and release, which would be helpful. Regards, Mark Walkom Infrastructure

Re: Recommended File System Settings for Elasticsearch

It really depends on your use, what you store and your queries. Reads would be random. Writes will depend on what you're doing, eg if you're doing logging then it will be mostly sequential. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web

Re: Cluster interface

Yep - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-network.html Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 18 July 2014 22:05, avery.ro...@insecure-it.com wrote: But all three

Re: Heap / GC Issues

How many indexes and how much data do you have? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 19 July 2014 01:09, Ned Campion nedcamp...@gmail.com wrote: Hey All, I've got a cluster with 5 data nodes (2 master

Re: Heap / GC Issues

How much in total, in the entire cluster. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 19 July 2014 10:55, Ned Campion nedcamp...@gmail.com wrote: Hey Mark, The index is currently 16GB with 10 indices (added

Re: Heap / GC Issues

You said you have one index of 16GB but that you have ~100 indexes in total. How much data across all those indexes, the cluster total in GB/TB. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 19 July 2014 11:25

Re: Kibana with (non-basic) User Authentication

There are a few such wrappers around that community members have written. Have a search through the archives here and you may get some ideas and even code to leverage. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com

Re: Cluster interface

ES needs direct access to the interface for the instance, so NAT won't work. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 18 July 2014 03:39, avery.ro...@insecure-it.com wrote: I've setup three kvm guests

Re: Any experience with ES and Data Compressing Filesystems?

There's a few previous threads on this topic in the archives, though I don't immediately recall seeing any performance metrics unfortunately. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 16 July 2014 20:56, horst

Re: How logs stored in Logstash/Elastisearch

1. It's indexed within Elasticsearch as a json document, one log entry in the Logstash is a document 2. The default is /var/lib/elasticsearch/data 3. No 4. You can backup using the snapshot API. What do you mean by remove and replace though? Regards, Mark Walkom Infrastructure Engineer Campaign

Re: How many tcp connections should ES/logstash generate ?

If you are using daily indexes then don't even bother running the delete, just drop the index when the next day rolls around. Resource temporarily unavailable could indicate you may need to increase the ulimit for the user, did you set this in /etc/default/elasticsearch? Regards, Mark Walkom

Re: High memory usage on dedicated master nodes

Are they master only or are you sending queries through them as well? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 17 July 2014 03:14, David Smith davidksmit...@gmail.com wrote: We have cluster with 22 data

Re: Get Perf Counters for ElasticSearch Nodes using JSON

You will want the cat API to start, then check out the cluster one next. http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index.html Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 17 July 2014

Re: Integration of latest Kibana 3.1 logstash 1.4.2 and elasticsearch 1.2.2 Integration

Yep, lots of people! Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 15 July 2014 17:21, Sandip Bankewar sbanke...@gmail.com wrote: Hi All, Has anyone Integration of latest Kibana 3.1 logstash 1.4.2

Re: tiering storage / Curator

There you go, I didn't know it did that! Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 15 July 2014 18:35, Patrick Proniewski elasticsea...@patpro.net wrote: It seems I can have multiple path.data on a single

Re: Integration of latest Kibana 3.1 logstash 1.4.2 and elasticsearch 1.2.2 Integration

/1.4.2/tutorials/getting-started-with-logstash Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 15 July 2014 19:29, Sandip Bankewar sbanke...@gmail.com wrote: Ohh Really... Could you please send me steps

Re: Issue Mapping Elasticsearch

. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 16 July 2014 07:15, Voc Austin voca...@gmail.com wrote: Hey, I am using ELK stack for log processing. Logstash 1.4.2 (Single Instance) and Elasticsearch 1.2.2(Cluster

<    4   5   6   7   8   9   10   11   12   13   >