Re: able to filter a field from document

On 10/06/2014 09:14 AM, David Pilato wrote: > I'm not sure I understood what you are looking for. > May be exclude is what you want? > > See: > http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-source-field.html#include-exclude This might have worked if Elasticsearch c

[hadoop] Issue while joining two hive tables stored on ES

Hi, I got an exception/error while joining two hive tables stored on ES. the tables I used are like below CREATE EXTERNAL TABLE member ( mem_idx bigint, id string, msaram_nm string, dept_nm string ) STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler' TBLPROPERTIES ( 'es.resource' = 'jdbc

Re: How to make Kibana Terms panel to count complete string frequencies, rather than separate words frequencies?

Does not it cause substantial inflation in the amount of data to be processed and stored at indexing time? As with most logs aggregation systems indexing is many orders of magnitude more frequent operation than querying and I'm concerned that using multi_fields instead of all simple string fiel

Re: How to make Kibana Terms panel to count complete string frequencies, rather than separate words frequencies?

I use multi fields to have several different analysis types supported as need and also to have the raw version available like in your example. On Monday, October 6, 2014 8:34:34 PM UTC-5, Konstantin Erman wrote: > I have documents in ES with the field "Message", which normally represents > som

[hadoop] Multiple indexes setting for 'es.resource'

Hi, I have a problem with 'es.resource' configuration for including multiple indexes. The hive table I created is like below CREATE EXTERNAL TABLE test ( date timestamp, clientip string, request string ) STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler' TBLPROPERTIES ( 'es.resource'

How to make Kibana Terms panel to count complete string frequencies, rather than separate words frequencies?

I have documents in ES with the field "Message", which normally represents some multi word text string. Trying to query it with Kibana to see which strings are in this property most frequently. What I actually get back is the table which shows frequency of the specific *words*, but not the whole

Re: Help with designing our document for graphs. Indexing single nodes in graph with thousands of incoming edges

Hi Jorg, Thanks for the response. I don't actually need to model the relationship per se, more that a document is used in a relationship via a filter, then search on it's properties. See the example below for more clarity. Restaurant: => {name: "duo"} Now, lets say I have 3 users, George, D

Re: Logstash into Elasticsearch Mapping Issues

I also have the following Logstash output configuration: output { # For testing only stdout { codec => rubydebug } # Elasticsearch via HTTP REST elasticsearch { protocol => "http" codec => json manage_template => false # Or whatever target ES host is required: ho

Re: Gibberish Classifier with Elastic Search - Is it Possible?

Lucene comes with some classifier implementations, but they are not integrated into Elasticsearch API yet. For more details, see http://soleami.com/blog/comparing-document-classification-functions-of-lucene-and-mahout.html Jörg On Mon, Oct 6, 2014 at 10:45 PM, Mauricio Moraes < mauricioflaviomo

Gibberish Classifier with Elastic Search - Is it Possible?

I have a system that stores user suggestions. Since the answers come from a touchscreen, the kids love to insert comments like: *asduhaduh3189189'uih'dh* *<3* *asdasdasdas* *gfdghgfhe* *n035gh9091v* I created a bayesian classifier and trained it with thousands of answers. Now it filters the us

Re: Logstash into Elasticsearch Mapping Issues

I haven't ever let logstash set the default mappings. Instead, whenever a logstash-style index is created, I let Elasticsearch set the default mappings from its template. That way, it works even if I replace logstash with something else. For example, with my $ES_CONFIG/templates/automap.json fi

Re: more accurate date based scoring

This is not Elasticsearch. Timestamps are represented in milliseconds as 64-bit longs. Scores are represented as 64-bit doubles. Per Java language specification, it is not possible to express a 64 bit long in a double because IEEE 754 only allows 52 bits. http://en.wikipedia.org/wiki/Double-preci

Re: Recommendation on reading the heart of the code.

There is no voting or other gamification, just a plain ol' mailing list. Many of us respond as just another way to contribute to open-source. -- Ivan On Mon, Oct 6, 2014 at 3:37 AM, ahmed jamal maaz wrote: > Hi all, > > These are very good advises. I really appreciate it. > > I am planning to

KeyError using python helpers.scan

Hi Everyone! I'm using ES for 6 months in production and today I was caught by an error being thrown by helpers.scan() method, what I could see is that scan always expect the response to contain the key _scroll_id, so I'm trying to understand it it's a problem in python wrappers or a problem in

more accurate date based scoring

PUT /test/test/1 { "date":"2013-04-01T00:00:00Z" } PUT /test/test/2 { "date":"2013-04-01T00:00:01Z" } PUT /test/test/3 { "date":"2013-04-01T00:00:03Z" } PUT /test/test/4 { "date":"2013-04-01T00:01:03Z" } Given these documents, I'm trying to come up with a query that scores them such th

Re: New version of Kibana in the works?

Thanks! Looks like Kibana 4.0.0-BETA1 is out: https://github.com/elasticsearch/kibana On Thu, Oct 2, 2014 at 11:38 PM, David Pilato wrote: > May be following elasticsearch Twitter account ? > > Envoyé de mon iPad > > Le 3 oct. 2014 à 07:55, Thibaut Labarre a écrit : > > Noted! > > What is the b

Optimal usage of big server (use VMs or not ...)

Regarding the recommendations for the maximal Java heap size, what is the best practice to utilize one server with lots of RAM (12 or 16 cores, 256 GB RAM): 1. Create 4 VMs, each 64 GB RAM and 32 GB Java heap size per VM - more shards per index 2. Use bare metal (256 GB RAM total), 128 GB Java

[ANN] Released trigram accelerated regex queries for Elasticsearch version 0.0.1

I just finished releasing the wikimedia extra Elasticsearch plugin which contains support for trigram accelerated regular expressions similar to PostgreSQL's implementation

'out of float scope for function score deviation' error using function_score and a gauss decay

Using elasticsearch 1.3.4 I have an index with user events and I'm trying to use function_score to get a sensible order by date (without using sort). The query below works, but only for relatively small result sets. While trying to test whether this orders things correctly (following up on a

[ANN] Released experimental highlighter version 0.0.12

I just finished releasing the experimental highlighter Elasticsearch plugin version 0.0.12. Its fixes one bug: * Regex highlighting fails on strings containing multi-byte characters If you

Trying to create geo_point index: Root type mapping not empty after parsing! Remaining field

Hi, I have an index with data that looks like this: { "id":"52534", "name":"foo", "location":{ "lat":39.18552295869955, "lon":-104.34581946342342 } } I'm attempting to make a m

Re: Marvel - monitoring multiple ES clusters by one Monitoring ES cluster

On Wednesday, October 1, 2014 2:59:00 PM UTC+2, W Koot wrote: > > Hi Boaz, > > Looking forward to the update. > I presume this will also address the "Warning: Multiple masters. 2 nodes > report master role" of the nodes panel? :-) > > If you monitor two clusters, you will have multiple masters -

Re: Need Help Regarding design multivalue column field in elasticsearch from river

The mail is hard to read, the layout is scrambled. For simple integer array, you do not need type "nested". What is the SQL statement you tried for JDBC river? You have seen the examples for bracket notation? Jörg On Mon, Oct 6, 2014 at 2:04 PM, wrote: > Hi All, > > I have a scenario like,

Cannot restore a snapshot with IndexMissingException[[_snapshot] missing]

We manage to occasionally put ES cluster into a particular state, where it would fail to restore index from a valid snapshot with 404 and: IndexMissingException[[_snapshot] missing] This is an exact quote, note '_snapshot', which is not the name of the index. Nothing in the main logs. The very

Re: Architecture to prevent slow queries

You could run less intense queries. Get more ram. Finally if io wait is a problem then you could switch to/add more solid state disks. Or you can add more nodes. We've done all of those for our Elasticsearch (no Logstash/Kibana in front though). Nik On Mon, Oct 6, 2014 at 4:43 AM, joergpra...

Need Help Regarding design multivalue column field in elasticsearch from river

Hi All, I have a scenario like, I need to build an index with four fields (Generic Description,GenericId ,GroupNo, List of Catalogsids ). Here is sample example of mysql table. Generic Description GenericIdGroupNo List of Catalogsids FrameKit 1 35 10,20,24,56,78,90,

Re: Error: No config files found: /etc/logstash/conf.d

On Monday, October 06, 2014 at 10:11 CEST, StueckJu wrote: > Hey, when i start logstash as a service /etc/init.d/logstash start, i > get the Error Message in the topic in file logstash.stdout. > I have a logstash config file in the directory > /etc/logstash/conf.d/server.conf. So i don't kno

Re: Recommendation on reading the heart of the code.

Hi all, These are very good advises. I really appreciate it. I am planning to start all of these in parallel and find which one will suite me the best. Thank you: Joerg, Isabel, Kevin and Ivan. I have a question: How do we vote an answer (like Quora) here (is this available with Google groups).

Mapping versioning by using doc_type

Hi, We've constantly facing with updates/changes in our document mapping during development, so we've come up with a idea of using `doc_type` for versioning. Basically we start from a `doc_type` like: `thing_0` as the start point. And each time we have some changes in the mapping, we created a

Re: Recommendation on reading the heart of the code.

A great start is by studying and writing plugins. Elasticsearch is one of the rare masterpieces of software that allow to plug in code that you have authored to extend functionality, without forking the main code base. There are a lot of plugins out there with small code base and easy to study, e

Re: Recommendation on reading the heart of the code.

When it comes to understanding any large code base your approach will always depend on your own preference. Things that I have found useful​ to get started: - Instead of working on the core inner workings of the code start with checking out the project, building it and using it. Build small demo a

Re: Architecture to prevent slow queries

Just add nodes. That's all :) Jörg On Mon, Oct 6, 2014 at 10:30 AM, Michael Irwin wrote: > Also, I would be grateful if someone could point me to some good general > information about this kind of thing. > > On Monday, October 6, 2014 4:28:31 AM UTC-4, Michael Irwin wrote: >> >> I'm using ES fo

Re: Architecture to prevent slow queries

Also, I would be grateful if someone could point me to some good general information about this kind of thing. On Monday, October 6, 2014 4:28:31 AM UTC-4, Michael Irwin wrote: > > I'm using ES for searching for events based on date and geo distance, as > well as textual content. I'm also using

Architecture to prevent slow queries

I'm using ES for searching for events based on date and geo distance, as well as textual content. I'm also using logstash for handling app logging and analytics. I've noticed after I have millions of records from logging/analytics, the events search starts slowing down. I'm currently using jus

Exists filter on fields with index_name not getting all documents

Hi guys, How to reproduce: https://gist.github.com/rmat0n/7b6f4f0398dc5f0966b4 I created 2 mapping with a testtext field on both, the difference is that each mapping have a different index_name for the field: the first mapping use 1.testtext and the second mapping use 2.testtext. Then putting 1

Error: No config files found: /etc/logstash/conf.d

Hey, when i start logstash as a service /etc/init.d/logstash start, i get the Error Message in the topic in file logstash.stdout. I have a logstash config file in the directory /etc/logstash/conf.d/server.conf. So i don't know why logstash doesnt find the file. Can anyone help me? Greets

Re: able to filter a field from document

I'm not sure I understood what you are looking for. May be exclude is what you want? See: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-source-field.html#include-exclude > Le 6 oct. 2014 à 09:02, Gurvinder Singh a > écrit : > > I am wondering if it is possibl

able to filter a field from document

I am wondering if it is possible to filter a field from a document using aliases. For example if I have an index with documents such as { "user" : "test", "secret_field": "value", "post_date" : "2014-06-05T09:12:12", "message" : "trying out Elasticsearch" } Now if I want to create