Re: Can someone point me to great live websites using ElasticSearch?

2015-05-31 Thread Mark Walkom
Wikipedia too ;) PS - We're moving to https://discuss.elastic.co/, please join us there for any future discussions! On 31 May 2015 at 05:01, Nikolas Everett nik9...@gmail.com wrote: Github. Stack overflow but their search isn't that nice the last time I checked. On May 30, 2015 2:53 PM,

Re: How do I index data from logs in a specific format in elastic search?

2015-05-28 Thread Mark Walkom
You can just send it as it and it will be indexed. Of course you can also map things so they are correctly identified. Depends on what you want to do. PS - We're moving to https://discuss.elastic.co/, please join us there for any future discussions! On 28 May 2015 at 23:20,

Re: Problem occured with same name of cluster in different machines

2015-05-28 Thread Mark Walkom
This is why - https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-zen.html#multicast Your best option is to use https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-allocation.html#shard-allocation-filtering and then move indices to their

Re: Where is Elasticsearch storing data

2015-05-28 Thread Mark Walkom
? If it helps, we are loading indexes from both hive and from logstash directly. Thanks, Jim On Wednesday, May 27, 2015 at 6:56:54 PM UTC-4, Mark Walkom wrote: Check out https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-dir-layout.html PS - We're moving to https

Re: Snapshot files are created every time even no change in index.

2015-05-28 Thread Mark Walkom
Snapshots are taken on the shard level, so if a shard changes due to things like merging, then it needs to record this. You can delete snapshots, ES will just hold onto the shards it needs. PS - We're moving to https://discuss.elastic.co/, please join us there for any future discussions! On 28

Re: Backup and Restore an index

2015-05-28 Thread Mark Walkom
Why not just use an alias, then you don't have to worry about conflicting names. Otherwise I have been having luck using LS 1.5 to reindex - https://gist.github.com/markwalkom/8a7201e3f6ea4354ae06 PS - We're moving to https://discuss.elastic.co/, please join us there for any future discussions!

Re: Elasticsearch Report Plugin to Generate Excel Reports

2015-05-28 Thread Mark Walkom
FYI rivers have been deprecated and will be removed from future versions of ES. While this plugin looks really handy, you may want to rearchitect it to take this into account :) PS - We're moving to https://discuss.elastic.co/, please join us there for any future discussions! On 29 May 2015 at

Re: Elasticsearch: 2-node cluster with failover

2015-05-28 Thread Mark Walkom
Master only nodes don't need to leverage FS caching as they hold no data. So you can easily increase their heap to 75% of system. However you really don't want to have less than 1GB of heap irrespective of the role it plays. PS - We're moving to https://discuss.elastic.co/, please join us there

Re: Considering scalability , is it right to keep a large number of primary shards at beginning?

2015-05-28 Thread Mark Walkom
You don't want 400 shards on 10 servers. You do want the ability to reindex to allow you to reshard to deal with this issue. Logstash 1.5 can do this very easily, see this example https://gist.github.com/markwalkom/8a7201e3f6ea4354ae06. However you probably don't want an index with 200 shards

Re: Considering scalability , is it right to keep a large number of primary shards at beginning?

2015-05-28 Thread Mark Walkom
. (・ˇ_ˇ・) 在 2015年5月28日星期四 UTC+8下午3:05:13,Mark Walkom写道: You don't want 400 shards on 10 servers. You do want the ability to reindex to allow you to reshard to deal with this issue. Logstash 1.5 can do this very easily, see this example https://gist.github.com/markwalkom/8a7201e3f6ea4354ae06

Re: Elasticsearch: 2-node cluster with failover

2015-05-28 Thread Mark Walkom
That's not enough heap, you want 1GB minimum. PS - We're moving to https://discuss.elastic.co/, please join us there for any future discussions! On 28 May 2015 at 20:25, prakhar prakhar.mishra1...@gmail.com wrote: Thanks for your reply. If I add one *master only* node on one of the two

Re: Not able to search Upper case values

2015-05-28 Thread Mark Walkom
You cannot set doc values on an analysed field. PS - We're moving to https://discuss.elastic.co/, please join us there for any future discussions! On 28 May 2015 at 20:03, vikas gopal vikas.ha...@gmail.com wrote: Thanks Allan, I have changed it to Customer: {type: string,index :

Re: Where is Elasticsearch storing data

2015-05-27 Thread Mark Walkom
Check out https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-dir-layout.html PS - We're moving to https://discuss.elastic.co/, please join us there for any future discussions! On 28 May 2015 at 01:27, David Pilato da...@pilato.fr wrote: by default in path.data dir. In zip

Re: Kibana 4 Process Crash

2015-05-26 Thread Mark Walkom
What do the logs show? PS - We're moving to https://discuss.elastic.co/, please join us there for any future discussions! On 26 May 2015 at 18:03, VV vincenzo.vign...@gmail.com wrote: Hello everyone I have a problem with Kibana 4, every moorning I open my browser on kibana page and I

Re: Indexing files (docx, xlsx etc)

2015-05-26 Thread Mark Walkom
For those you will want to use something like Apache Tika, there is nothing in the ELK stack to do these sorts of files. PS - We're moving to https://discuss.elastic.co/, please join us there for any future discussions! On 26 May 2015 at 17:47, Gary Wenneker g...@gary.nu wrote: Is there a way

Re: Problems with plugins (Kopf)

2015-05-26 Thread Mark Walkom
Try adding another / to the end - http://localhost:9200/_plugin/kopf/ On 26 May 2015 at 22:34, Invest Ict i...@investict.nl wrote: I have some troubles with my indexes, so i wanted to install a plugin named Kopf. So i downloaded the zip file from GIT and placed it on the server. Next thing i

Re: ProcessClusterEventTimeoutException in Elasticsearch. Is this timeout value configurable? If yes how?

2015-05-25 Thread Mark Walkom
This thread is nearly 18 months old, you should really create a new one with a little more detail. PS - We're moving to https://discuss.elastic.co/, please join us there for any future discussions! On 26 May 2015 at 04:36, mzrth_7810 afrazmam...@gmail.com wrote: Could someone please give an

Re: elasticsearch curator 3.0.3: option --older-than

2015-05-20 Thread Mark Walkom
Try curator --host localhost delete indices --older-than 1 --time-unit days --prefix .marvel --timestring %Y.%m.%d PS - We're moving to https://discuss.elastic.co/, please join us there for any future discussions! On 21 May 2015 at 06:58, Jean-Max Reymond jmreym...@gmail.com wrote: I have

Re: Modeling Objects That Have Many Properties

2015-05-15 Thread Mark Walkom
You probably want to use parent/child instead, that way you don't have to update the entire document each time, just the child(ren). PS - We're moving to https://discuss.elastic.co/, please join us there for any future discussions! On 15 May 2015 at 22:52, Peter Hodgman peter.hodg...@gmail.com

Re: Daily index with static mapping

2015-05-14 Thread Mark Walkom
] update_mapping [INDEX_TYPE1] (dynamic) Any idea about what could be the missing part ? BR. On Sunday, 10 May 2015 23:37:06 UTC+2, Mark Walkom wrote: You want to look at http://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html Also, the link you provided

Re: extract/export Kibana charts to embed/inport them somewhere else

2015-05-14 Thread Mark Walkom
-graphics-in-enduser-projects/482 On Thursday, May 14, 2015 at 1:41:09 AM UTC+2, Mark Walkom wrote: You can embed KB4 into external web based apps. But you cannot with KB3. PS - We're moving to https://discuss.elastic.co/, please join us there for any future discussions! On 14 May 2015 at 08

Re: Reindex into another Elasticsearch

2015-05-14 Thread Mark Walkom
and outputting to our ES cluster - That's our answer to what we were looking for Att Frederico Ferreira (21) 98714-1445 2015-04-27 18:28 GMT-03:00 Mark Walkom markwal...@gmail.com: 1 shard per index doesn't make a lot of sense unless you have very small amounts of data, You'd be better

Re: extract/export Kibana charts to embed/inport them somewhere else

2015-05-13 Thread Mark Walkom
You can embed KB4 into external web based apps. But you cannot with KB3. PS - We're moving to https://discuss.elastic.co/, please join us there for any future discussions! On 14 May 2015 at 08:38, Luca G. Soave luca.so...@gmail.com wrote: Is there a convenient way of snapshotting Kibana

Re: Missing operating systems statistics on 2 nodes out of 5

2015-05-12 Thread Mark Walkom
Are you using containers, Docker, LXC etc? On 12 May 2015 at 18:22, Alexander Vassilevski alexander.vassilev...@gmail.com wrote: Hey guys, I'm sorry if this is a duplicate post - I did in fact do various searches in the group prior to writing this, but I'd be more than glad to use a URL if

Re: Elastic Search configuration

2015-05-12 Thread Mark Walkom
configuration is feasible or not and what will happen if i bring down shard to 1. My ES version is 1.5.2 and java release is 8.45 On Tuesday, May 12, 2015 at 3:38:40 PM UTC+5:30, Mark Walkom wrote: Are you having performance issues now? If so I'd start with replacing your disk setup, chances

Re: Elastic Search configuration

2015-05-12 Thread Mark Walkom
Are you having performance issues now? If so I'd start with replacing your disk setup, chances are it's not as fast as you think, especially if you are storing the data from both nodes on the same path. Also what ES version are you on, what Java release and version? On 12 May 2015 at 19:01,

Re: Elasticsearch hardware requirement,and benchmarking

2015-05-11 Thread Mark Walkom
. Regards, On Monday, May 11, 2015 at 4:43:17 AM UTC+7, Mark Walkom wrote: As I said, we don't recommend sending queries, which includes those generated by Kibana, to master only nodes. You would be better off sending them to data nodes. As for your performance problems, that's a multi

Re: Troubleshooting ES Resharding. Nature of immediate tasks and other questions

2015-05-11 Thread Mark Walkom
that some of the current indexes have update-mapping operations frequently applied to it, as the document structure might have new fields. Thanks for your time! Ale El lunes, 4 de mayo de 2015, 22:01:46 (UTC-3), Mark Walkom escribió: The rationale of queuing is to allow for instances where

Re: Daily index with static mapping

2015-05-10 Thread Mark Walkom
You want to look at http://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html Also, the link you provided earlier is from 0.90.X, which is now really old. If you are on that version it's best to upgrade. PS - We're moving to https://discuss.elastic.co/, please join us

Re: Elasticsearch takes a very long time to stop

2015-05-10 Thread Mark Walkom
1. Yes 2. That doesn't sound right, I'd suggest upgrading to 1.5.X and see if the problem still exists and then we can go from there. It may mean raising an issue on Github to track down a problem. Though the methods you have listed are the best to stop the process. PS - We're

Re: Elasticsearch hardware requirement,and benchmarking

2015-05-10 Thread Mark Walkom
As I said, we don't recommend sending queries, which includes those generated by Kibana, to master only nodes. You would be better off sending them to data nodes. As for your performance problems, that's a multi-layered problem that may not be solved just by adding more nodes. You need to provide

Re: AutoScale elasticsearch

2015-05-10 Thread Mark Walkom
You can certainly use AWS autoscaling groups to accomplish things, but there is nothing native in ES to do it. On 11 May 2015 at 12:07, John S bun...@gmail.com wrote: Hi, Currently am having a cluster with 4 nodes provisioned in Aws Ec2. Occasionally there were some issue happened like disk

Re: Elasticsearch hardware requirement,and benchmarking

2015-05-10 Thread Mark Walkom
​None of the nodes *needs* super high performance hardware, but it will definitely benefit from it. Given the ubiquity of gigabit ethernet, you shouldn't be going for any less. 10GBe is nice if you can afford it. Master only nodes don't need to be large, they can be simply a few CPU cores, a few

Re: Multiple nodes on same machine : replicas?

2015-05-06 Thread Mark Walkom
Have you tried upgrading? That's a super old version which may not help. Though that is the right setting. On 06/05/2015 3:24 pm, DH ciddp...@gmail.com wrote: Hello, everyone. I'm trying to set up a cluster of 10 nodes, distributed on five different computers. Each computer have 2 full

Re: kibana

2015-05-06 Thread Mark Walkom
​Kibana provides unrestricted access to any index it can see. So unless you are running Shield to limit that, or use aliases, then it's possible users can see data they should not be able to see.​ ​ ​There is no function​ality to restrict specific queries though. On 7 May 2015 at 01:16, Lior

Re: question on keeping separate marvel monitoring cluster

2015-05-06 Thread Mark Walkom
It can be one node, Marvel does not create much data. That node needs to be a master and a data node. On 6 May 2015 at 20:56, Darshat Shah dars...@gmail.com wrote: Hi, The Marvel installation guide indicates that in production we should deploy marvel on 2 different nodes in a different

Re: Memory usage of the machine with ES is continuously increasing

2015-05-06 Thread Mark Walkom
When the underlying lucene engine interacts with a segment the OS will leverage free system RAM and keep that segment in memory. However Elasticsearch/lucene has no way to control of OS level caches. What exactly is the problem here? This caching is what helps provide performance for ES. ​ --

Re: ES upgrade

2015-05-04 Thread Mark Walkom
1.7.0_75-b13) Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode) On Saturday, May 2, 2015 at 2:36:31 AM UTC+5:30, Mark Walkom wrote: Are you running the same ES and java versions on all your nodes and clients? On 1 May 2015 at 21:21, phani.n...@goktree.com wrote: Hi All i

Re: About replica shard use many field data

2015-05-04 Thread Mark Walkom
Replica's don't double the amount of field data loaded. A query will only load what it needs from the shards it needs, but it won't load primary and replica, just one or the other. Ideally you should 1) upgrade to aggs, facets are deprecated and not as performant, and 2) upgrade to doc_values.

Re: Elasticsearch crashing, Dashboards missing

2015-05-04 Thread Mark Walkom
Kibana dashboards are saved in kibana-int for KB 3 and .kibana for KB4. It looks like you are running out of heap, ie an OOM. How much memory have you assigned to ES and can you increase that? On 4 May 2015 at 18:12, xiro86 amir.gli...@gmx.at wrote: Hi, I've set up ELK-Stack for Monitoring

Re: ES upgrade

2015-05-04 Thread Mark Walkom
I'd try disabling your firewall and eliminating it as a problem, checking if the nodes can reach each other, and then going from there. On 4 May 2015 at 20:23, phani.nadimi...@goktree.com wrote: mark I opened that port already even though I am getting that warning... [2015-05-04

Re: Kibana, Error editing saved visualization with quotationmark

2015-05-04 Thread Mark Walkom
You might have to extract the doc from the .kibana index, edit it and then send it back to get immediate access. However it'd also be worth raising this as a Github issue, perhaps there is something that is not being properly escaped under the hood, which causes the behaviour. On 4 May 2015 at

Re: Troubleshooting ES Resharding. Nature of immediate tasks and other questions

2015-05-04 Thread Mark Walkom
The rationale of queuing is to allow for instances where temporary load on the cluster might otherwise reject a request. There is no way to prioritise tasks over other tasks. Though it looks like your problem is you are overloading your nodes. 32192 primary shards is a massive amount for only 12

Re: Dynamically setting discovery.zen.ping.unicast.hosts

2015-05-04 Thread Mark Walkom
This is not a dynamic setting, it needs to be defined in your config file and ES needs to be restarted to read it. On 5 May 2015 at 08:03, Satnam Singh satnam6...@gmail.com wrote: I would like to dynamically set discovery.zen.ping.multicast.enabled and discovery.zen.ping.unicast.hosts -- is

Re: Creating and storing ES indices on S3

2015-05-04 Thread Mark Walkom
It might be possible, but not really recommended as I'd imagine it'd be pretty slow, which would impact your performance of ES. On 4 May 2015 at 23:56, Lavesh Gupta lavesh.gu...@druva.com wrote: Hi Everyone, I went through the group and came across various replies by David Pilato, Kimchy

Re: Split brain problem in 2 node elasticsearch cluster

2015-05-03 Thread Mark Walkom
1. Why are they becoming split anyway? GC, other load, network? 2. Not if they both think they are masters. 3. Are you running replicas? If so ES doesn't really differentiate between the two. On 4 May 2015 at 15:03, Gourav H Dhelaria gouravdhela...@gmail.com wrote: Version: 1.4.

Re: ES upgrade

2015-05-01 Thread Mark Walkom
Are you running the same ES and java versions on all your nodes and clients? On 1 May 2015 at 21:21, phani.nadimi...@goktree.com wrote: Hi All i upgraded elastic search from 1.4.2 to 1.5.2 i am getting following warning from console after upgrade please explain me the following error

Re: Kibana - Is there such a thing as a permanent filter?

2015-04-30 Thread Mark Walkom
Not at this stage, this is something that is being worked on for future releases. On 1 May 2015 at 03:22, Andrew Ruslander andrew.ruslan...@gmail.com wrote: I have a system that multiple users use. I want to make a single Kibana dashboard that the users could pull up, but it would only show

Re: My elasticsearch powered grafana server gives me org.apache.lucene.index.CorruptIndexException exceptions when I try to restore from backup. Help!

2015-04-30 Thread Mark Walkom
Grafana uses graphite, do you mean Kibana? On 1 May 2015 at 15:09, jblock jbl...@lyft.com wrote: Had a problem with my grafana server today so I am now forced to restore from backups. When I try to restore from any of my many backups, I get exceptions such as this: Caused by:

Re: Failed to get setting group for [threadpool.] setting prefix and setting [threadpool.bulk] because of a missing '.'

2015-04-30 Thread Mark Walkom
How are you setting this? On 30 April 2015 at 22:02, marc.fal...@happn.com wrote: Hello there I noticed the following warning message in my nodes logs when starting: [2015-04-30 12:40:13,265][INFO ][node ] [***server***] started [2015-04-30 12:40:32,154][WARN

Re: How to set Marvel license programmatically in Ansible deploy?

2015-04-30 Thread Mark Walkom
Just to add, the shield repo isn't publically accessible do to it being one of our commercial plugins, so I can't link you to the issue sorry! On 1 May 2015 at 07:37, Mark Walkom markwal...@gmail.com wrote: Unfortunately there isn't a way to do this, but I've raised a feature request to get

Re: How to set Marvel license programmatically in Ansible deploy?

2015-04-30 Thread Mark Walkom
Unfortunately there isn't a way to do this, but I've raised a feature request to get this added. On 30 April 2015 at 23:39, Joel Potischman joel.potisch...@beatport.com wrote: We have a license for Marvel and while the documentation just says to enter it through the Marvel UI when prompted, we

Re: installation procedure of ES

2015-04-29 Thread Mark Walkom
It's bad practice to run things as root that don't need to, there's a while bunch of security concerns there. On 29 April 2015 at 20:58, phani.nadimi...@goktree.com wrote: Hi All, i have small question can we install elasticsearch with out creating elastic search user in linux systems.I

Re: Is external Load Balance useful?

2015-04-29 Thread Mark Walkom
It'd help if you had client nodes. But it also helps if you don't, even though ES does connect to the nodes with the shards it needs, distributing the initial query load across multiple nodes means that you don't have a single node that has to deal with all the indexing and querying results from

Re: Is external Load Balance useful?

2015-04-29 Thread Mark Walkom
at 3:27:42 PM UTC+8, Mark Walkom wrote: It'd help if you had client nodes. But it also helps if you don't, even though ES does connect to the nodes with the shards it needs, distributing the initial query load across multiple nodes means that you don't have a single node that has to deal

Re: where is compatibility matrix?

2015-04-29 Thread Mark Walkom
KB3 will work with any recent version of ES. So will LS if you use the HTTP protocol. On 30 April 2015 at 06:18, Sitka sitkaw...@gmail.com wrote: I need to move backward from the current release of ELK due to inability to get Kibana to load pages. There used to be a compatibility matrix

Re: matching threat indicators to content in ELK

2015-04-29 Thread Mark Walkom
Why not use the translate filter in LS to add a field based on these IPs (eg alertableIP: true) and then run a percolator query? On 30 April 2015 at 01:24, Chris Adams chrisadam...@gmail.com wrote: Folks... Looking for suggestions here on approach for a simple use case. I have ELK up and

Re: possible networking problem?

2015-04-28 Thread Mark Walkom
What do your Kibana logs show? On 29 April 2015 at 07:53, Sitka sitkaw...@gmail.com wrote: I have installed elasticsearch and kibana. I started elasticsearch and did a GET to test it out. Everything worked. I installed kibana next. When I test it doing http://localhost:5061; it fails. I

Re: Cluster falling into YELLOW state during use of Snapshot API

2015-04-28 Thread Mark Walkom
What do your ES logs show? On 29 April 2015 at 10:02, Steven B sbancr...@expectlabs.com wrote: Our Elasticsearch cluster is comprised of 4 very large instances running Elasticsearch 1.4.0. The Snapshot API is set up to take backups once a day in the evening. We maintain 5 snapshots at a

Re: possible networking problem?

2015-04-28 Thread Mark Walkom
KB4 logs to stdout, so once you start the binary you should see lots of output of in your command prompt. On 29 April 2015 at 09:23, Colleen Roe sitkaw...@gmail.com wrote: I've searched the Kibana installation directories and don't see any log files. On Tue, Apr 28, 2015 at 3:39 PM, Mark

Re: More memory or more CPU cores help better performance?

2015-04-28 Thread Mark Walkom
Depends - you will want to do some tests to see what sort of resources your use case requires. Start with smaller machines and go from there. On 29 April 2015 at 12:17, Xudong You xudong@gmail.com wrote: hi, I am building ES on cloud Virtual machines, the cloud platform provides different

Re: mlockall needs to be docuemented for systemd

2015-04-28 Thread Mark Walkom
This has been raised in https://github.com/elastic/elasticsearch/issues/9357 On 29 April 2015 at 05:04, Karl Putland k...@simplesignal.com wrote: drwxr-xr-x. 26 root root 12288 Apr 28 14:57 . [root@node8 system]# diff -u elasticsearch.service.2015-04-28\@14\:57~ elasticsearch.service ---

Re: Elasticsearch puppet module's problem

2015-04-28 Thread Mark Walkom
You don't want to set that in the init script, use the init_defaults hash instead. On 29 April 2015 at 00:09, Sergey Zemlyanoy s.zemlya...@gmail.com wrote: And guys, how can I increase heap size using this module? I don't see how to control parameter ES_HEAP_SIZE located in init script

Re: The best way to migrate data from 3 indices to new index

2015-04-27 Thread Mark Walkom
Use one of the official clients to do it - http://www.elastic.co/guide/ On 27 April 2015 at 20:02, tao hiko taoh...@gmail.com wrote: I have similar 3 indices in same cluster and I will create new index for support new change. Anyone please suggest me, How to migrate data from those to the

Re: Storage vs index size, memory foot print etc.

2015-04-27 Thread Mark Walkom
You don't need to store the entire index in memory, that's now how ES works. Regarding the second point, you index a document's fields which allows you to search it, storing means you can also return the value of the field if it is found in the search. On 27 April 2015 at 23:23, Jakub Stransky

Re: Kibana 4 initial configuration doesn't pick up the expected indices

2015-04-26 Thread Mark Walkom
You cannot use wild card prefixes. There is functionally no difference between * and *-something, as each index has to be checked to see if it matches, so you might as well just use *. On 27 April 2015 at 12:19, Amos S amos.shap...@gmail.com wrote: Hi, I'm installing Kibana 4.0.0 on Linux

Re: Issue when MatchPhasePrefix and Sort

2015-04-24 Thread Mark Walkom
See http://www.elastic.co/guide/en/elasticsearch/guide/master/_limiting_memory_usage.html#circuit-breaker Basically ES is protecting you against potential OOM killers. On 25 April 2015 at 01:27, TB txind...@gmail.com wrote: The field is not of 594 MB, could this be related to where the JVM

Re: Elasticsearch crashed after start

2015-04-23 Thread Mark Walkom
What do the logs show? On 24 April 2015 at 12:03, Ann Yablunovskaya lad.sh...@gmail.com wrote: Hi! I don't understand, what happend. OS CentOS 7.1 I have ES cluster with two servers. It have the same configuration. I tried to configure shield and marvel but my second ES instanse have

Re: Suspicious connections on ES

2015-04-22 Thread Mark Walkom
iptables -I INPUT -p tcp -s my_ip --dport 9200:9400 -j ACCEPT $ sudo iptables -P INPUT -j DROP ``` I forgot to say that I set `script.disable_dynamic: false` to run some external js scripts. At that time, ES was still v1.3.7. On Thursday, April 23, 2015 at 8:57:42 AM UTC+8, Mark Walkom wrote

Re: Is there a way to know memory required

2015-04-22 Thread Mark Walkom
UTC-4, Mark Walkom wrote: Not really; how large are your fields, are they analysed, are you using doc values? You really need to test this with your own data set. On 18 April 2015 at 15:40, bvnrwork budda...@gmail.com wrote: Hi , Is there a way to know minimum memory required for X

Re: Grouping/extracting results uploaded to Elasticsearch

2015-04-22 Thread Mark Walkom
You ideally want to restructure your data and split the fields out. If you can't do it in your code then Logstash would be able to do something. On 22 April 2015 at 23:05, KT SSP kevin.to...@ssp-uk.com wrote: Hello We have a build process using ant that is externally monitored (web page)

Re: Elasticsearch puppet module's problem

2015-04-22 Thread Mark Walkom
The module works with the notion of an instance https://github.com/elastic/puppet-elasticsearch/blob/master/README.md#instances and when you setup an instance it creates /etc/init.d/elasticsearch-$instancename On 22 April 2015 at 17:42, Sergey Zemlyanoy s.zemlya...@gmail.com wrote: Update

Re: Suspicious connections on ES

2015-04-22 Thread Mark Walkom
Is your ES instance open to the world? Check your ES logs as well. On 22/04/2015 8:44 pm, Jason Zhang moc...@gmail.com wrote: Also, I've noticed there're many suspicious files in /tmp, like: ``` $ ls -al /tmp 26000 32 991linux conf.n elasticsearch/ gates.lock git icp Intelip

Re: 30 billion unique documents (and counting)

2015-04-22 Thread Mark Walkom
. Is that right? On Wed, Apr 22, 2015 at 2:56 PM, Mark Walkom markwal...@gmail.com wrote: If you are using time series data then you should be using time series indices. As Fred pointed out, routing an entire month's worth of data to a single shard is not going to scale. Also, we recommend that you keep

Re: 30 billion unique documents (and counting)

2015-04-22 Thread Mark Walkom
If you are using time series data then you should be using time series indices. As Fred pointed out, routing an entire month's worth of data to a single shard is not going to scale. Also, we recommend that you keep shard size below 50GB, this helps with recovery and distribution. There is also a

Re: Data too large error

2015-04-22 Thread Mark Walkom
This original thread is nearly a year old! You'd be better off starting a new one :) On 23 April 2015 at 01:36, Spencer Owen owenspen...@gmail.com wrote: Did you figure this out? I'm running into the same problem. On Thursday, July 31, 2014 at 3:22:28 AM UTC-6, Rhys Campbell wrote: I

Re: Rebuilding master node caused data loss

2015-04-21 Thread Mark Walkom
Default is still yes. What happened in the logs on the data nodes? On 22 April 2015 at 00:23, Brian brian.sand...@gmail.com wrote: I have a cluster with 5 data nodes, and 1 master node. I decided to test a master node failure, and clearly I miss understood exactly what is stored on the

Re: deploying ElasticSearch to a large memory server

2015-04-21 Thread Mark Walkom
It's definitely reasonable to run multiple instances per physical here. On 21 April 2015 at 19:22, Tzahi jakubovitz tza...@hotmail.com wrote: Hi all, I have a server with 1.5 TB memory. I can either use it with a single ES process, or launch few separate instances (using either VM, docker,

Re: selecting a server - a single quad socket, or two dual socket

2015-04-21 Thread Mark Walkom
It may make sense to do this, you probably also want to look into running multiple instances on the host to maximise capacity. On 21 April 2015 at 19:08, Tzahi jakubovitz tza...@hotmail.com wrote: Today we can buy very performant servers at very reasonable price points. e.g. – the price of

Re: How to export dashboard and visualization by using elasticdump?

2015-04-21 Thread Mark Walkom
Take a look at https://github.com/taskrabbit/elasticsearch-dump On 21 April 2015 at 23:25, Priya G g.shanmugapriy...@gmail.com wrote: Can anyone tell me the steps how to install elasticdump and how to export and import dashboards? -- You received this message because you are subscribed to

Re: Elasticsearch service often goes down or gets killed

2015-04-21 Thread Mark Walkom
You need to monitor the cluster with something like Marvel, kopf or HQ to find out what is happening. ES may die if the cluster is overloaded (think OOM), but you should see something in the logs on that. On 20 April 2015 at 22:13, Sébastien Vassaux svass...@gmail.com wrote: Hello! My

Re: Document getting lost

2015-04-21 Thread Mark Walkom
Simply increasing the depths means more things will queue, but you still need to catch up on that queue. If you are overloaded then this will never happen and your queue won't be much help. Look at the larger picture, are you running out of resources consistently or is it transitory? On 22 April

Re: Very sluggish Elasticsearch node; not sure why

2015-04-19 Thread Mark Walkom
I'd guess that you are hitting the capacity of the node. Try closing/deleting indices or upgrading the instance. On 19 April 2015 at 13:03, Dave Galbraith david92galbra...@gmail.com wrote: So I've got this Elasticsearch single-node instance running out on an EC2 m3.2xlarge in the cloud. I'm

Re: equivalent of sql join and update

2015-04-19 Thread Mark Walkom
Logstash has a translate filter - http://www.elastic.co/guide/en/logstash/current/plugins-filters-translate.html However KB3 has this functionality built in, where you can change a displayed value to something else, it'll be released in KB4 soon. On 19 April 2015 at 22:11, dna lor

Re: creation_date in index setteing

2015-04-19 Thread Mark Walkom
It's when the index was created, it's unix epoch time. On 20 April 2015 at 14:06, tao hiko taoh...@gmail.com wrote: I query setting information of index and found that have creation_date field but I cannot understand what is value. Can you explain me more? settings: { index:

Re: Access to specific kibana dashboards

2015-04-18 Thread Mark Walkom
In this case, for example, if I want to restrict an URL like http://myESHost:9200/_plugin/kopf/#/!/cluster, what do I have to put after Location /???. Sorry if I have asked a very naive question. Thanks again for your time. Cheers! Ruby On Friday, April 17, 2015 at 12:23:50 AM UTC+2, Mark Walkom

Re: Is there a way to know memory required

2015-04-18 Thread Mark Walkom
Not really; how large are your fields, are they analysed, are you using doc values? You really need to test this with your own data set. On 18 April 2015 at 15:40, bvnrwork budda08n...@gmail.com wrote: Hi , Is there a way to know minimum memory required for X no of fields created in index ?

Re: Date model for elastic search document

2015-04-18 Thread Mark Walkom
Look at nested docs, where you have the static part which is the parent, and then any dynamic data in the child. http://www.elastic.co/guide/en/elasticsearch/guide/current/nested-objects.html On 18 April 2015 at 15:33, bvnrwork budda08n...@gmail.com wrote: Can some one help me in designing

Re: 1.5.1 upgrade failure

2015-04-17 Thread Mark Walkom
Can you provide a bit more of the log? This may imply corruption but it's hard to tell without context. On 17 April 2015 at 01:32, Ted Smith tedsmithgr...@gmail.com wrote: Hi, I just upgraded from 1.5.0 to 1.5.1 I got bunches of errors with following I think show the issue [nested:

Re: Could use some help with using Doc Values

2015-04-17 Thread Mark Walkom
. On 18 April 2015 at 12:32, Scott Chapman scottedchap...@gmail.com wrote: Thanks Mark. Exactly what I was looking for. Once I make the change is there any way I can tell it is being used properly for a specific field? On Friday, April 17, 2015 at 10:23:15 PM UTC-4, Mark Walkom wrote: You can add

Re: Could use some help with using Doc Values

2015-04-17 Thread Mark Walkom
isn't explicitly in the template. What would it look like? Also, once I have made the change to my template, what's the right way to test it (validate that for a new index i is using Doc Value for the specific field)? On Thursday, April 16, 2015 at 7:57:41 PM UTC-4, Mark Walkom wrote: As per

Re: Deleted index directories still can data logs through kibana

2015-04-16 Thread Mark Walkom
Elasticsearch Curator (https://github.com/elasticsearch/curator) is a better way to manage deletion of indices. Deleting them off the file system is messy. On 16 April 2015 at 16:50, Ch Ravikishore ravikishore.ris...@gmail.com wrote: Hi, I deleted the index directories from

Re: Access to specific kibana dashboards

2015-04-16 Thread Mark Walkom
You could do this with apache/nginx ACLs as KB3 simply loads a path, either a file from the server's FS or from ES. If you load it up you will see it in the URL. On 16 April 2015 at 21:58, Rubaiyat Islam Sadat rubaiyatislam.sa...@gmail.com wrote: Hi all, As a completely newbie here, I am

Re: How to configure max file descriptors on windows OS?

2015-04-16 Thread Mark Walkom
-1 means unbound, ie unlimited. On 16 April 2015 at 20:54, Xudong You xudong@gmail.com wrote: Anyone knows how to change the max_file_descriptors on windows? I built ES cluster on Windows and got following process information: max_file_descriptors : -1, open_file_descriptors : -1,

Re: Saturating the management thread pool

2015-04-16 Thread Mark Walkom
Also related https://github.com/elastic/elasticsearch/issues/10447 On 17 April 2015 at 12:37, Charlie Moad charlie.m...@geofeedia.com wrote: This was tracked down to a problem with Ubuntu 14.04 running under Xen (in AWS). The latest kernel in Ubuntu resolves the problem, so I had to do a

Re: Could use some help with using Doc Values

2015-04-16 Thread Mark Walkom
the structure of the template should be... Thanks! On Wednesday, April 15, 2015 at 8:29:28 PM UTC-4, Mark Walkom wrote: Yes that is correct, you have to update your mappings and wait for new indices to be created from it, it's not something that can be applied retroactively without reindexing

Re: Could use some help with using Doc Values

2015-04-15 Thread Mark Walkom
PM UTC-4, Mark Walkom wrote: Start here and you'll be good to go - http://www.elastic.co/guide/ en/elasticsearch/guide/current/doc-values.html On 16 April 2015 at 08:03, Scott Chapman scotted...@gmail.com wrote: Probably. I just need some help figuring out how to do that. Help? On Wednesday

Re: Searches slow down significantly for several seconds every minute with transport client

2015-04-14 Thread Mark Walkom
Have you checked the logs for GC events or similar? What about the web logs for events coming in? On 15 April 2015 at 09:03, Daryl Robbins darylrobb...@gmail.com wrote: I am seeing a consistent bottleneck in requests (taking about 2+ seconds) at the same second every minute across all four of

Re: Please suggest.

2015-04-14 Thread Mark Walkom
And what do you want help on? On 15 April 2015 at 00:38, vikas gopal vikas.ha...@gmail.com wrote: Hi Experts, Need your valuable suggestion here . I have following setup and it is working fine. 1) 2-ES nodes (in Cluster handled by AWS load balancer) 2) 1-LS node (push data to

Re: Kibana 4 - ability to select a date range on dashboard that is reflected in other visualizations

2015-04-14 Thread Mark Walkom
The other visualisations should definitely update. I'd suggest you might need more data to show. On 15 April 2015 at 00:36, Thomas Bratt thomas.br...@gmail.com wrote: Hi, I am using Kibana 4 with a Date Histogram. I can select a time range with the mouse but the other visualizations on the

  1   2   3   4   5   6   7   8   9   10   >