Re: indexing- pls help

You could give a look at FSRiver plugin. Might help to start. https://github.com/dadoonet/fsriver David > Le 20 févr. 2015 à 06:42, aiswarya lakshmi a écrit : > > Hi.. > > I am a beginner here.. I just installed elasticsearch on windows and i added > the sense extension for google chrome..

Disk awarnes on Indexing.

Hi All, Few of our nodes filled disks up to 90%. Is there any option to avoid indexing on those disks instead of relocating shards. Because each shard contains roughly 600G data, so relocating is costly as ours is very busy cluster. Thanks Prasath Rajan -- You received this message becaus

Re: Combining Multiple Queries with 'OR' or 'AND'

Hi, bool query should work: - { "query": { "bool": { "should": [ { "filtered": { // Query1 "query": {...}, "filter": {...} } }, { "filtered": { // Query2 "q

Re: Elasticsearch Script merge the results of two aggregations

Hi, Looks like you are using lucene expression [1]. See the link for the limitation of lucene expression. Today it only supports numeric values. Since terms agg doesn’t have lang property, probably you have “script.default_lang" set to “expression" in elasticsearch.yml? FYI, if you put “lang”:”

indexing- pls help

Hi.. I am a beginner here.. I just installed elasticsearch on windows and i added the sense extension for google chrome.. I need to index some files that are present in my local system directory.. I did not install any extension other than sense.. Please tell me what more should i install

Re: Elasticsearch performance tuning

I listed below instance and his heap size details. Medium instance 3.75 RAM 1 cores Storage :4 GB SSD 64-bit Network Java heap size: 2gb R3 Large 15.25 RAM 2 cores Storage :32 GB SSD Java heap size: 7gb R3 High-Memory Extra Large r3.xlarge 30.5 RAM 4 cores Java heap size: 15gb Thanks

Re: Elasticsearch Broken?

Your curl is incorrect when using a file. IIRC you should do something like: curl -XPUT "http://localhost:9200/(index name)" -d @mapping.json But check curl help. -- David ;-) Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > Le 20 févr. 2015 à 02:49, Cong Hui a écrit : > > Hello, >

Java node client failing to send join request to master

Anyone have any idea where to start with this one? We're running our singular master/data node in a docker container, trying to connect via a Java node client on different boxes. Please help!! INFO [2015-02-20 02:55:11,411] org.elasticsearch.discovery.ec2: [localhost] failed to send join r

Elasticsearch Broken?

Hello, I recently started attempting to use elasticsearch to store some data. In the learning process, I was doing ok until I attempted using the bulk api. I used the something along the lines of: conn = ES(url, timeout, bulksize) for each (tuple) data = something(tuple) conn.index

Search multiple indices in Kibana 4?

Hi all, Trying out Kibana 4's new release today, and was wondering is this is still possible. In Kibana 3, you could simply comma-delimit all the index patterns you wanted to expose to your searches, but that doesn't seem possible in Kibana 4. I have indexes named like: company-customer-MMD

Combining Multiple Queries with 'OR' or 'AND'

Hi, Question I am trying to combine 2 user search queries using AND and OR as operation. I am aware of combining queries where we can merge filters, but I want to merge entire queries like { {BIG Elastic Query1} AND {BIG Elastic Query2} }. Details For instance say a user performs a search for

Re: elastic search cluster behind azure public load balncer

Do not run ES across regions, it's a bad idea. Use snapshot and restore or some other method to replicate data. On 19 February 2015 at 20:51, Subodh Patil wrote: > Ok Thanks Mark. > So you are saying I should set node.master: true as well as node.data: > true for all the 3 nodes in the cluste

Re: Incorrect results from geo_shape filter

FYI, you can also try a search using these coordinates (which are clearly incorrect) in your filter, and it still returns a result. "coordinates": [ -119, 35 ] On Thursday, February 19, 2015 at 3:04:29 PM UTC-8, Alexei Peters wrote: > > Hey Everyone, > > I have a nested document that includes

Incorrect results from geo_shape filter

Hey Everyone, I have a nested document that includes a spatial component that I'm using in a spatial query via the geo_shape filter. I've noticed that the filter consistently returns a result that includes a geometry that clearly falls outside the spatial filter. I've tested the spatial intersec

Re: What tools to use for indexing ES in production?

Kafka is fine, you could also use Logstash to read this data and send to ES - http://www.elasticsearch.org/guide/en/logstash/current/plugins-inputs-kafka.html NB this is in the 1.5 BETA release, so use with caution. Otherwise take a look at other Logstash inputs and see if there is something suita

Re: Move Data from One index to Another Cluster Index

The python client has a reindex helper that can do just that - just supply a client instance for the source and destination clusters. http://elasticsearch-py.readthedocs.org/en/latest/helpers.html#elasticsearch.helpers.reindex Hope this helps, Honza On Thu, Feb 19, 2015 at 11:44 PM, Amay Patil

Re: Help with 4 node cluster

You can have 4 master eligable ​if you want​ , just set discovery.zen.minimum_master_nodes ​ to 3 to ensure a quorum.​ But ideally as Christian mentioned, it's best to have an uneven number of masters. On 19 February 2015 at 16:51, wrote: > Hi, > > You always want an odd number of master nodes

Re: Elasticsearch performance tuning

Don't change cache and buffer sizes unless you know what is happening, the defaults are going to be fine. How much heap did you give ES? I'm not sure you can do much about the date filter though, maybe someone else has pointers. On 19 February 2015 at 21:12, Deva Raj wrote: > Hi Mark Walkom, >

Move Data from One index to Another Cluster Index

How can one move data from One index in a server to another Index in another Server. I know reindexing moves data into another index in the same cluster. How can it be done in Python ? Ap -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To

Re: elastic search on t2.micro (Amazon WS)

Except if you're running ES along with Couchbase you really want more RAM anyway, 1GB for both is difficult. On 20 February 2015 at 09:01, wrote: > What's your heap size (JVM setting) set to? > > It needs to be at most one half the machine's RAM, so set it to 500MB. > > J > > On Wednesday, Febru

Re: Elasticsearch index name question

This is possible but not automatically within ES. LS knows it needs to switch to a new index at UTC, you need to find a way to get the river or some other code to do this. On 19 February 2015 at 21:59, Silvana Vezzoli wrote: > I use a Monitoring Framework designed as a solution to monitor >

Re: elastic search on t2.micro (Amazon WS)

What's your heap size (JVM setting) set to? It needs to be at most one half the machine's RAM, so set it to 500MB. J On Wednesday, February 18, 2015 at 3:52:22 PM UTC+2, Seung Chan Lim wrote: > > I'm trying to see if I can get elastic search (ES) 1.3.8 working with > couchbase (CB) 3.0.2 on a t

how to avoid MapperParsingException?

Hi, I'm indexing documents with the following structure: { "name": "peter", "email": "p...@p.com", "location": "MIA" } { "name": "mary", "email": "m...@m.com", "device": "ipad" } { "name": "mary", "email": "m...@m.com", "metadata": { ... } } As you can see, I only know the type for "name" and "e

Re: formula or guidelines to calculate/estimate the index size

There is nothing official. Just create an index, put in 1 of your documents and then extrapolate. On 20 February 2015 at 04:26, Gaurav gupta wrote: > Could anyone assist me to know how can I find a formula or guidelines to > calculate/estimate the index size created by elastic search. I fou

[Spark] Unable to index JSON from HDFS using SchemaRDD.saveToES()

This is my first real attempt at spark/scala so be gentle. I have a file called test.json on HDFS that I'm trying to read and index using Spark. I'm able to read the file via SQLContext.jsonFile() but when I try to use SchemaRDD.saveToEs() I get an invalid JSON fragment received error. I'm th

Re: elastic search on t2.micro (Amazon WS)

Depends on your dataset and use. I don't go below 2GB heaps when I'm just testing things. On 20 February 2015 at 05:52, Seung Chan Lim wrote: > What's the minimum RAM requirement? > > slim > > On Wednesday, February 18, 2015 at 5:18:58 PM UTC-5, Mark Walkom wrote: >> >> Your only real option her

elasticsearch-http-basic with ES 1.4.2

Has anyone had any luck with using http-basic with ES 1.4.2? I just want to put some basic security on my ES instance from outside of the clusters and this appears to be the easiest way with just white listing my other nodes. When I install it and configure it, it shows it going to the http-basi

Multi Level nested search by using NEST API

I am having trouble to make multi level nested query by using NEST API. Here is my mapping. { "log": { "mappings":{ "LogEvent": { "properties":{ "@timestamp":{"type":"date","store":true,"format":"-MM-dd'T'HH:mm:ss"},

Re: elastic search on t2.micro (Amazon WS)

What's the minimum RAM requirement? slim On Wednesday, February 18, 2015 at 5:18:58 PM UTC-5, Mark Walkom wrote: > > Your only real option here is to get a machine with more RAM. > Try spinning up a VM locally, on your desk/laptop. > > On 19 February 2015 at 00:52, Seung Chan Lim > wrote: > >> I

Re: waited for 30s and no initial state was set by the discovery

Additional information - we can telnet into or ES server from our application server on 9300. On Thursday, February 19, 2015 at 10:04:51 AM UTC-8, Diana Tuck wrote: > > I know other people have posted this, but I've tried everything that the > other threads have said to try. We keep getting thi

Re: Cluster hanging on node failure

I posted here too: http://stackoverflow.com/questions/28601885/cluster-hanging-on-node-failure Would love to get some help with this. Best, Max Den onsdag 18 februari 2015 kl. 20:30:46 UTC+1 skrev Max Charas: > > Hello all of you bright people, > > We’re currently running a smallish 300 GB clust

Re: ClassNotFoundException: org.elasticsearch.discovery.ec2.Ec2DiscoveryModule

Thanks, David. I eventually ended up finding the pom on the github repo. Thanks for adding the documentation!! On Wednesday, February 18, 2015 at 10:39:19 PM UTC-8, David Pilato wrote: > > Yes. > > This should be added to the doc: > https://github.com/elasticsearch/elasticsearch-cloud-aws/issu

waited for 30s and no initial state was set by the discovery

I know other people have posted this, but I've tried everything that the other threads have said to try. We keep getting this at startup of our java node client: INFO [2015-02-19 17:57:45,206] org.elasticsearch.node: [localhost] version[1.4.2], pid[11103], build[927caff/2014-12-16T14:11:12Z]

formula or guidelines to calculate/estimate the index size

Could anyone assist me to know how can I find a formula or guidelines to calculate/estimate the index size created by elastic search. I found that there is a formula (excel sheet) for it for LUCENE. Thanks! Gaurav -- You received this message because you are subscribed to the Google Groups "ela

Re: [Hadoop][Spark] Exclude metadata fields from _source

Thanks for the response Costin! As you mentioned, option 1, i.e es.mapping.exclude, is more appropriate when working with JSON. Since it doesn't seem to work, I've followed your advice and raised a new issue (https://github.com/elasticsearch/elasticsearch-hadoop/issues/381) including a small tes

Re: I want to implement QueryElevationComponent feature in ELasticSearch

I have implemented query elevation by a function score-based conditional boosting plugin, see https://github.com/jprante/elasticsearch-functionscore-conditionalboost Jörg On Thu, Feb 19, 2015 at 2:40 PM, Gurunath pai wrote: > HI All, > > I want to implement QueryElevationComponent feature in E

Using 'elapsed' plugin with GrayLog2

I am working with a team who are creating a proof of concept monitoring and analytics system using a combination of LogStash, MongoDB, Elasticsearch and GrayLog2. I was wondering if anyone has any experience using the elapsed plugin with this set up and can tell me what (if any) issues they en

disappearing log records: what can I do?

hello group, I have the following stack: fluentd (td-agent 2.1.3), elasticsearch (1.4.4), kibana (3.1.2) to manage the logging of an in-company application. At first glance, this seems to work OK, but it appears that from time to time, the reported records in kibana don't match the line count of

How to index multiple database query data to a single index file in Elastic search.

Hi, Want to index multiple database data to Elastic search, I have done the same in Solr using DataImportHandler and DeltaImportHandler. Is there any way to achieve the same in Elastic Search, also want to index all these multiple query output all at once. I have observed that we cant index mu

I want to implement QueryElevationComponent feature in ELasticSearch

HI All, I want to implement QueryElevationComponent feature in ELasticSearch, Can anyone suggest me how can I go ahead with this feature in Elastic Search. Thanks Guru Pai. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from

Re: problem with using uax_url_email

Hi, for people having the same problem like me, here an answer I received from Pablo in PT group: About your problem I beleive this is a constraint of the Apache Tika [1], which is used by the mapper-attachment plugin. I believe that a search over Tika pdf limitations or a question on their li

Re: Elasticsearch Script merge the results of two aggregations

I second error : { "error": "SearchPhaseExecutionException[Failed to execute phase [query_fetch], all shards failed; shardFailures {[g][test][0]: RemoteTransportException[[Mammomax][inet[/192.168.1.8:9300]][search/phase/query+fetch]]; nested: SearchParseException[[test][0]: query[ConstantScore(

[ES-1.4.0] Snapshot Queue

Hi All, Is there a way to queue snapshot invocations? e.g.: snapshot_1 -> snapshot_2 -> -> snapshot_N Thanks, Yarden -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send

Elasticsearch Script merge the results of two aggregations

I use aggregrations on elasticsearch version 1.3.8. I use aggregation script for awhile today ıt didnt work. Please help ı cant find any solution : This the mapping: "mappings": { "product": { "properties": { "brandId": { "type": "in

Elasticsearch index name question

I use a Monitoring Framework designed as a solution to monitor heterogeneous networks and systems in terms of services (platforms, applications for TELCO systems). This framework collects in synchronous way the required data from several devices, it stores them in a Mongo Data Base and then t

Script based transform during index

Hi Group First apologies if this is not the right way to ask the below question but this is my first time. I have some documents with source IP and destination IP address. I want to enhance these documents with geo info with script transform when they arrive. So I created a script in python and

Re: Elasticsearch performance tuning

Hi Mark Walkom, I have given below logstash conf file Logstash conf input { file { } } filter { mutate { gsub => ["message", "\n", " "] } mutate { gsub => ["message", "\t", " "] } multiline { pattern => "^ " what => "previous" } grok { match =

Re: elastic search cluster behind azure public load balncer

Ok Thanks Mark. So you are saying I should set node.master: true as well as node.data: true for all the 3 nodes in the cluster. And this will hold true for 4 node or even more node cluster ? As of now I am making data redundant on all 3 nodes but please suggest performance... azure Load ba

Date Histogram Bucket Count ?

Is it possible to apply a filter to date histogram buckets ? For example return only buckets that are below a certain value. I was looking to do it with scripts but dont know how to access buckets in script, so if anyone knows anything about that, it would be very very very helpful -- You rece

Re: ES OOMing and not triggering cache circuit breakers, using LocalManualCache

After some experimentation, I believe _cluster/stats shows the total field data across the whole cluster. I manged to push my test cluster to 198MiB field data cache usage. As a result, based on Zachary's feedback, I've set the following values in my elasticsearch.yml: indices.fielddata.cache.

Re: Aggregations failing on fields with custom analyzer..

I understand what you are saying.. I was able to recreate the same error you showed myself.. I was not able to insert into your index whose mapping is "double", but I am able to insert a string into my older index whose mapping is "double".. Very weird.. But I don't know how you could recreate

Re: What tools to use for indexing ES in production?

Well, the sales messages are coming through kafka. We need to extract some info from the database. We can do anything really. I'm just not sure what are the common practice here. It seems to be so many options. what kind of questions am I not asking here? On Monday, February 16, 2015 at 2:30:4

Re: ElasticSearch search performance question

Good stuff. You're seeing the benefits of not caching lots of single-use BitSets. Now if you swap queries for your filters then you'll also see the benefits of not allocating multi-megabyte BitSets to hold what is typically a single bit for each query that you run. On Thursday, February 19, 201

Re: Aggregations failing on fields with custom analyzer..

If you can provide a full example working as I did, we can try it and see what is wrong. -- David Pilato | Technical Advocate | Elasticsearch.com @dadoonet | @elasticsearchfr | @scrutmydocs

Re: Is it safe to change node names in an existing ElasticSearch cluster

Ok, thanks again. 2015-02-19 9:06 GMT+01:00 David Pilato : > Yes. > > David > > Le 19 févr. 2015 à 08:56, Jan-Erik Westlund a > écrit : > > Correct, in that case, it will not be a rolling upgrade ;-) The service > will be down for a few minutes. > Can I then change all the nodenames, and the sta

Re: Aggregations failing on fields with custom analyzer..

I"m getting this error as well using your PUT requests.. It feels like I'm doing something wrong.. But I don't know what exactly.. I'm using this index template.. https://gist.github.com/syllogismos/c2dde4f097fea149e1a0 I didn't specify a particular mapping from my index but reindexed from a p

Re: Aggregations failing on fields with custom analyzer..

I think you are doing something wrong. DELETE index PUT index { "mappings": { "doc": { "properties": { "foo": { "type": "double" } } } } } PUT index/doc/1 { "foo": "bar" } gives: { "error": "MapperParsingException[failed to parse [foo]]; nes

Re: Aggregations failing on fields with custom analyzer..

"_source" : { "Sort" : "", "gt" : "2015-02-18T15:07:10", "uid" : "54867dc55b482b04da7f23d8", "usId" : "54867dc55b482b04da7f23d7", "ut" : "2015-02-18T20:37:10", "act" : "productlisting", "st" : "2015-02-18T15:07:46", "Filter" : "", "av" : "3.0.0.0", "ViewType" : "SmallSingleList", "os" : "Windows",

Re: Aggregations failing on fields with custom analyzer..

I don’t know without a concrete example. I’d say that if you map have a type number and you send "123" it could work. -- David Pilato | Technical Advocate | Elasticsearch.com @dadoonet | @elasticsearchfr | @scrutmydocs

Re: Aggregations failing on fields with custom analyzer..

It was my mistake, the field I was trying to do an aggregation was mapped double, I assumed its a string, after seeing some sample documents with strings.. Why didn't es throw an error when I'm indexing docs with strings instead of double..? On Thursday, February 19, 2015 at 1:35:08 PM UTC+5:3

Re: Is it safe to change node names in an existing ElasticSearch cluster

Yes. David > Le 19 févr. 2015 à 08:56, Jan-Erik Westlund a écrit : > > Correct, in that case, it will not be a rolling upgrade ;-) The service will > be down for a few minutes. > Can I then change all the nodenames, and the start the services on all the > nodes with the new names without mes

Re: Aggregations failing on fields with custom analyzer..

Did you apply your analyzer to your mapping? David > Le 19 févr. 2015 à 08:53, Anil Karaka a écrit : > > http://stackoverflow.com/questions/28601082/terms-aggregation-failing-on-string-fields-with-a-custom-analyzer-in-elasticsear > > Posted in stack over flow as well.. > >> On Thursday, Febru