Re: Shield: Why do I get timeout on my own node

As pointed out by Jay Modi when discussing another question I had, this is a timeout issue. The pi's do not have a lot of processing power and adding ssl is to heavy for the pi. Upgrading the timeout to 30 seconds did help. On Thu, Feb 12, 2015 at 9:41 PM, Jettro Coenradie < jettro.coenra...@gmail

Re: Tips for failing handshake with shield

Hi Jay, thanks again for the help. You were so right. Change the timeout to 30s and now the cluster is up and running. Now I can go for the next steps. regards Jettro On Fri, Feb 13, 2015 at 3:12 AM, Jay Modi wrote: > I think you are running into the TLS handshake taking too long and it > cause

Re: Hibernate Search with Elasticsearch

Ok. But still the injection works? So you can start from here. The REST part should not be a concern here as unrelated to your original question. David > Le 13 févr. 2015 à 08:07, William Yang a écrit : > > I ran this part in cmd and nothing happened after I did jetty:run in Spring > for bran

Re: Hibernate Search with Elasticsearch

I ran this part in cmd and nothing happened after I did jetty:run in Spring for branch00 # Create one person curl -XPUT http://127.0.0.1:8080/api/1/person/1 -d '{"name":"David Pilato"}' # Read that person curl http://127.0.0.1:8080/api/1/person/1 -- You received this message because you are subs

Re: Hibernate Search with Elasticsearch

I meant "what exact curl command you run?" David > Le 13 févr. 2015 à 07:17, William Yang a écrit : > > I ran jetty:run on branch00. > > I think that happened because when I used curl -XPUT, nothing happened. > However, when I initialized a database and it produced results, I was able to > us

Re: Hibernate Search with Elasticsearch

I ran jetty:run on branch00. I think that happened because when I used curl -XPUT, nothing happened. However, when I initialized a database and it produced results, I was able to use the search function. On Thu, Feb 12, 2015 at 9:57 PM, David Pilato wrote: > CRUD=Create, Read, Update, Delete >

Re: Hibernate Search with Elasticsearch

CRUD=Create, Read, Update, Delete No. What exact operation did you run? David > Le 13 févr. 2015 à 01:22, William Yang a écrit : > > Ah I see, I will try to go through the other branches as well. > > However, when I try to do the CRUD operations (What does that stand for by > the way?), I g

mysql to elasticsearch replication using elasticsearch-river-jdbc

Hi all, I am working on mysql to ES replication using elasticsearch-river-jdbc.my configuration is, curl -XPUT '10.10.10.53:9200/_river/test/_meta' -d '{ "type" : "jdbc", "jdbc" : { "driver" : "com.mysql.jdbc.Driver", "url" : "jdbc:mysql://localhost:3306/Mahesh", "user" : "root

customized analyzers behave

Hi All, I am trying to write down a custom analyzer like as follows : "analysis": { "tokenizer": { "my_edge_ngram_tokenizer": { "type": "edgeNGram", "min_gram": "1", "max_gram": "20", "token_chars": [] } },

Kibana authentication proxy?

I want authentication for my elastic-search cluster so that security is handled. So I came across Kibana authentication proxy So I installed it and got the app.js running. But there are no instructions on how to use it to achieve authenti

Re: Single low memory data node impact whole cluster

Thanks for the reply. Today the problem happened again. A bad node stop responding and bring down the whole cluster. But this time memory is ok. Here are some details. 1. Again, management APIs such as _node, cat are not returning. Default the default 9200 response. If I directly hit the master nod

Re: Tips for failing handshake with shield

I think you are running into the TLS handshake taking too long and it causes the pings to timeout and then the Netty channels are closed, which is why you see the connection closed messages. Since you are running on a Raspberry PI, the overhead of TLS is going to be a lot for that platform; the

Re: Is Unsecured OPTIONS Method A Vulnerability?

A quick check shows that ES returns nothing with an options request via curl. ES uses netty to serve HTTP content. On 12 February 2015 at 08:50, wrote: > I've been working lately on a project utilizing ElasticSearch and Kibana. > To secure the ElasticSearch API I've hidden it behind a reverse p

Re: function score filter function doesn't seem to be cached

set explain to true and read the explain output if cache is use? Jason On Fri, Feb 13, 2015 at 3:47 AM, wrote: > I am using a function score with a function filter to score a subset of > results higher. I specify to cache the function filter, but its performance > is slower and slower the more

Re: Hibernate Search with Elasticsearch

Ah I see, I will try to go through the other branches as well. However, when I try to do the CRUD operations (What does that stand for by the way?), I get a 16:20:55,055 DEBUG [PersonRestAPI] create(1, ) 16:20:55,055 DEBUG [PersonService] get(1)=null Do you know the cause of that? -- Yo

Re: ElasticSearch search performance question

You might want to try hitting hot threads while putting your load on it and seeing what you see. Or posting it. Nik On Thu, Feb 12, 2015 at 4:44 PM, Jay Danielian wrote: > Mark, > > Thanks for the initial reply. Yes, your assumption about these things > being very specific and thus not likely

Re: Master Node vs. Data Node Architecture

Depending why the node goes down - going mid-way with dedicated master nodes is sometimes the solution And if this is due to massive use of aggregations, doc-values may be the answer (or larger heap, but that's costlier) -- Itamar Syn-Hershko http://code972.com | @synhershko

Re: Is it possible to delete data/field without affecting the index?

Thanks a lot, that sounds exactly like what I was looking for! Why would you suggest extracting the content myself? Because of the "experimental" state of the attachment type plugin? Even if I'd extract the content myself I wouldn't want to store it in ES (as I'd never request it from ES). The on

Re: Is it possible to delete data/field without affecting the index?

You could may be use source exclude: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-source-field.html#include-exclude Though I think it would be better to extract yourself content using Tika if you are using Java and only send what you need to ES. David > Le 12

Re: Moving to production - Storage and servers

SAN and NAS are in the same boat, using network storage with ES is problematic. DAS and JBOD are fine. ES will run fine on hardware or virtualised. It really comes down to your needs and budget. On 13 February 2015 at 01:55, wrote: > Hi everyone, > > After my first >

Re: Multy-tenancy - level of service garantee

How do you guarantee a level of service provided any other way? Redundancy and smart planning and design. It's no different with ES. On 13 February 2015 at 01:47, wrote: > Hi everyone, > > After my precedent question >

Re: Updating several indices at once

You can use wildcards, you can also comma separate them eg - curl -XPUT 'localhost:9200/logstash-2014.01.01,logstash-2014.01.02/_settings On 12 February 2015 at 23:53, Aldian wrote: > Hi > > Today I tried the following request, which worked great: > curl -XPUT 'localhost:9200/logstash-2014.03.24

Re: Master Node vs. Data Node Architecture

Except that is overkill when you only have 3 nodes. How much data do you have in the cluster? On 13 February 2015 at 01:15, Itamar Syn-Hershko wrote: > See this: > http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-node.html > > Basically, the recommended pattern talk

Is it possible to delete data/field without affecting the index?

Hi, I have some terabytes of documents (pdf, office, etc) stored in some system outside of ES. Suppose I want to make them *searchable* with ES, however I will never serve the original documents from ES, but from that other system. Is it possible to send the documents to ES (e.g. via base64 enco

Re: ES logs and cluster health

You'd have to ask the author of head why it doesn't align. Does ES have permissions to write to the new log directory? On 13 February 2015 at 05:47, caspertz wrote: > We have a 4 node ES cluster running version 1.4.1. We are using Ubuntu > and have it set up as a service. We have defined our

Re: ElasticSearch search performance question

It'd help if you could gist/pastebin/etc a query example. Also your current ES and java need updating, there are known issues with java <1.7u55, and you will always see performance boosts running the latest version of ES. That aside, what is your current resource utilisation like? Are you seeing

ElasticSearch search performance question

Term filters are useful for clauses that have a high hit rate in terms of reuse across queries. Given you are searching for what I imagine are unique records using low-frequency terms I expect the following to be true of the bitsets being cached by these filters: 1) they are rarely reused and th

Re: A strange behavior we've encountered on our ELK

Well SSD would also fix all the pains for my bank too... (-; Are you sure it's caused by disk latency and not some sort of mis-tuned TCP driver? I've read some blogs that recommeded to increase some of the buffers at the sysctl.conf. Do you think so too? On Thursday, February 12, 2015, Itamar Syn

Shield: Why do I get timeout on my own node

Hi I am trying to create a cluster that is secured on a number of raspberry pi's. If I start just one node with the following config I get a timeout to that same node. So it is trying to ping itself and that does not work. Any ideas? Config: cluster.name: jc-mypi node.name: jc-pi-red marvel.age

ElasticSearch search performance question

I know this is difficult to answer, the real answer is always "It Depends" :) But I am going to go ahead and hope I get some feedback here. We are mainly using ES to issue terms searches against fields that are non-analyzed. We are using ES like a key value store, where once the match is found

Re: Hibernate Search with Elasticsearch

Well. I thought you asked me to run the example I wrote. If so, the example uses in 00-legacy branch a database, insert, get, delete (CRUD) and search using the database (with hibernate). Branch 01-direct adds elasticsearch for the search part. Hibernate and the database is still used for CRUD op

Re: Hibernate Search with Elasticsearch

I still need this line even though I want to replace Hibernate with ElasticSearch? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@go

Re: Hibernate Search with Elasticsearch

I think you need to modify this line as well: https://github.com/dadoonet/legacy-search/blob/00-legacy/src/main/webapp/WEB-INF/person-dao-context.xml#L29 -- David Pilato | Technical Ad

function score filter function doesn't seem to be cached

I am using a function score with a function filter to score a subset of results higher. I specify to cache the function filter, but its performance is slower and slower the more clauses I add to the filter. A query with 2 geo_polygons clauses returns relatively fast (30ms), but the more polygon

Re: Hibernate Search with Elasticsearch

Hi David, I managed to figure out the MySQL things! I'm now getting these errors and I don't know why. I'm testing only an empty database called "test", there aren't any tables in it: 11:43:10,397 ERROR [SchemaExport] Unsuccessful: alter table Person add constraint FK8E4887752ACD7745 foreign

Re: Hibernate Search with Elasticsearch

David, I fixed the dependency problem - couldn't find the empty-99 commons-logging. However, I'm getting this now: 11:18:46,164 WARN [JDBCExceptionReporter] SQL Error: 0, SQLState: null 11:18:46,164 ERROR [JDBCExceptionReporter] Cannot create PoolableConnectionFactory (Connection refused. C

Re: Tips for failing handshake with shield

Thanks Jay for your answer. Had a good look at the appendix with troubleshooting, but cannot really find something to help me out. On the client I see this message: [2015-02-12 20:10:28,363][ERROR][shield.transport.netty ] [jc-pi-glas] SSL/TLS handshake failed, closing channel: null There is a

Re: Hibernate Search with Elasticsearch

Hi again David, The error that I get when trying to do jetty:run is: [ERROR] Failed to execute goal on project legacy-search: Could not resolve dependencies for project fr.pilato.demo:legacy-search:war:1.0-SNAPSHOT: Could not find artifact commons-logging:commons-logging:jar:99-empty in ce

ES logs and cluster health

We have a 4 node ES cluster running version 1.4.1. We are using Ubuntu and have it set up as a service. We have defined our location for logs as a specific directory other than /var/log/elasticsearch which is the default. When this starts, the process shows the value we set for the log area

Re: Search in plugin hangs

Can you publish your code so it can be reproduced? Then you might get feedback. Jörg On Thu, Feb 12, 2015 at 12:58 PM, Mauricio Scheffer < mauricioschef...@gmail.com> wrote: > I got this working by creating my own TransportClient instance instead of > using the injected Client. > Still, it would

Re: A strange behavior we've encountered on our ELK

Well SSD would also fix all the pains for my bank too... (-; Are you sure it's caused by disk latency and not some sort of mis-tuned TCP driver? I've read some blogs that recommeded to increase some of the buffers at the sysctl.conf. Do you think so too? On Thursday, February 12, 2015, Itamar Syn

Re: real one shot jdbc river

Yes, a river instance is restarted by ES automatically at node restart. You must delete it after work is done in order to free resources. Just a note, in future versions, I will drop the ES river API, and the JDBC plugin will manage an internal state, where the number of execution cycles can be co

Re: real one shot jdbc river

So I tried the oneshot strategy with no success. The river is being rerun after restart of cluster. Am Donnerstag, 12. Februar 2015 15:22:27 UTC+1 schrieb Abid Hussain: > > Hi all, > > we run a jdbc river once per night for a complete recreation of our index. > > The river - once it has been crea

Re: Elasticsearch + attachment plugin + Kibana + couchbase

Yes, that too :) Also if its a time based data, you will not be able to use kibana's date filtering etc - because it lacks the @timestamp field. Basically, the XDCR elasticsearch plugin was built around the XDCR / Couchbase realm and not around Elasticsearch's. Unfortunately this means many ES fe

Re: Elasticsearch + attachment plugin + Kibana + couchbase

ok, ill try. this "envelope document", is it something i need to be concerned about when I'm querying via Kibana? On Thursday, February 12, 2015 at 5:14:39 PM UTC+2, Itamar Syn-Hershko wrote: > > Yes. Just make sure the template reflects the actual document structure - > as I said XDCR wraps y

Re: ES OOMing and not triggering cache circuit breakers, using LocalManualCache

Oh, is field data per-node or total across the cluster? I grabbed a test cluster with two data nodes, and I deliberately set fielddata really low: indices.fielddata.cache.size: "100mb" However, after a few queries, I'm seeing more than 100MiB in use: $ curl "http://localhost:9200/_cluster/stats

Re: Elasticsearch + attachment plugin + Kibana + couchbase

Yes. Just make sure the template reflects the actual document structure - as I said XDCR wraps your document in an envelope document -- Itamar Syn-Hershko http://code972.com | @synhershko Freelance Developer & Consultant Lucene.NET committer and PMC member On Thu

Re: Elasticsearch + attachment plugin + Kibana + couchbase

ok, just to be clear. the steps i did was as followed: 1. create the index with the mapping. 2. define the XDCR to replicate my bucket with the index in ES. 3. insert data to couchbase. 4. try to query with kibana What you suggest is to Add another BEFORE step 1: 0. create a template to include

Re: Elasticsearch + attachment plugin + Kibana + couchbase

Like I said, you need the mapping to "catch" before the XDCR plugin begins the replication - so you need to put a template with this mapping that will override XDCR's -- Itamar Syn-Hershko http://code972.com | @synhershko Freelance Developer & Consultant Lucene.NE

Moving to production - Storage and servers

Hi everyone, After my first and second question regarding clusters' examples and multy-tenancy with garantee of service, I have one mor

Re: Elasticsearch + attachment plugin + Kibana + couchbase

Thanks you for the response i am using mapping, i created the following index PUT /storage/files/_mapping { "files": { "properties": { "file": { "type": "attachment", "path": "full", "fields": { "content_type": { "type": "string",

Multy-tenancy - level of service garantee

Hi everyone, After my precedent question regarding examples of clusters in production, I am wondering about multy-tenancy and garantee of service in Elasticsearch : *Multy-tenant cluster* : Is there a way to *garant

Use cases - Production examples: datas, queries, cluster hardware and conf, and statistics

Hi everyone, I am considering moving one or several elasticsearch clusters to production. Although Elasticsearch's documentation and community is *great*, I am strongly startled not to find any *complete use-case story* stretching from application(s) needs and data considerations to hardware one

Moving to production considerations - applications with their clusters' infrastructure

Hi everyone, I am considering moving one or several elasticsearch clusters to production. Although Elasticsearch's documentation and community is *great*, I am strongly startled not to find any *complete use-case story* stretching from application(s) needs and data considerations to hardware one

real one shot jdbc river

Hi all, we run a jdbc river once per night for a complete recreation of our index. The river - once it has been created - will run again after a restart of elasticsearch cluster (single node cluster). But what we try to achieve is a one shot river, which isn't being re-run after restart of clus

Re: Master Node vs. Data Node Architecture

See this: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-node.html Basically, the recommended pattern talks about isolating responsibilities. A node should either be a data node, master-eligible node, or an external gateway to the cluster (client node) -- Itamar Sy

Master Node vs. Data Node Architecture

Hello, Currently I have a 3 node ElasticSearch cluster. Each node is a RHEL VM with 16 gig RAM. The basic config is: - All nodes can be master and are data nodes. - 3 shards and 1 replica - 6 different indexes I'm starting to run into issues of ElasticSearch bogging down on searches and is com

Re: Elasticsearch + attachment plugin + Kibana + couchbase

The XDCR plugin indexes the data using an envelope document. Long story short, make sure you use the latest XDCR plugin as older ones are missing lots of important functions, and use templates and dynamic templates with proper field paths for this to work correctly http://code972.com/blog/2015/02/

Elasticsearch + attachment plugin + Kibana + couchbase

Hi, I'm new to the group, hope ill find what i need and share my experience as i go along.. im using ES with the attachment-plugin in order to store and search files. when i set the mapping right and insert the file data in a Base64 manner I'm able to query my data via Kibana. my problem is t

Re: Delete index after backup

Found the last section here 'archiving old indices' to answer my question. On Wednesday, February 11, 2015 at 5:17:04 PM UTC+2, Yarden Bar wrote: > > but will the index deletion get reflected in the next sna

Re: Different ES clusters in the same host..

Hello David, It's really solve my problem ;) Thanks Ricardo Reis Em sexta-feira, 6 de fevereiro de 2015 08:52:03 UTC-2, David Pilato escreveu: > > What we do this in AWS plugin: > > - get the IP address from AWS API (which obviously does know anything > about the port used) > address = instan

Re: A strange behavior we've encountered on our ELK

There's a good writeup on the subject by Mike btw, you should read it http://www.elasticsearch.org/blog/performance-considerations-elasticsearch-indexing/ -- Itamar Syn-Hershko http://code972.com | @synhershko Freelance Developer & Consultant Lucene.NET committer

Re: A strange behavior we've encountered on our ELK

Yes, make sure the disk is local and not low latency shared one (e.g. SAN). Also SSD will probably fix all your pains. -- Itamar Syn-Hershko http://code972.com | @synhershko Freelance Developer & Consultant Lucene.NET committer and PMC member On Thu, Feb 12, 2015

Re: A strange behavior we've encountered on our ELK

Sort of... The ELK is running as a VM on a dedicated ESXi. Are there special configurations I should do in such a case? Thanks, Yuval. On Thursday, February 12, 2015, Itamar Syn-Hershko wrote: > Yes - can you try using the bulk API? Also, are you running on a cloud > server? > > -- > > Itamar S

Updating several indices at once

Hi Today I tried the following request, which worked great: curl -XPUT 'localhost:9200/logstash-2014.03.24/_settings' -d ' { "index" : { "number_of_replicas" : 0 } }' Now I want to set number of replicas for every indice in 2014 to 0. Can I use a wildcard in the url such as the f

Re: performance getting even worse after optimization

Hi Xiaoting Ye, Do you find a solution for your problem? Best, On Thursday, January 8, 2015 at 9:39:16 PM UTC+3:30, Xiaoting Ye wrote: > > The index is of 149G, 19 shards with 1 replica. > > The es version is 1.4.1, and the java version is 1.7.0_71. > > I have specific routing strategy and the

Re: Search in plugin hangs

I got this working by creating my own TransportClient instance instead of using the injected Client. Still, it would be nice to understand what's going on here, also locking up the node like this seems like a pretty serious bug. On Tuesday, February 10, 2015 at 12:27:07 PM UTC, Mauricio Scheffer

Re: A strange behavior we've encountered on our ELK

Yes - can you try using the bulk API? Also, are you running on a cloud server? -- Itamar Syn-Hershko http://code972.com | @synhershko Freelance Developer & Consultant Lucene.NET committer and PMC member On Thu, Feb 12, 2015 at 11:28 AM, Yuval Khalifa wrote: > H

Our approach to make custom facet's order

Please see below { "aggs": { "category": { "terms": { "field": "category.name", "order": { "sorting": "asc" } }, "aggs": { "sorting": { "avg": { "field": "category.sortCode" } } } }

node.js client timeout doesn't work (for me)

hi guys, i wonder if someone had the same problem. i have a heavy query, with complicated script, and i want to give this query larger timeout. but no matter, what number i put there, the timeout happens after 30 seconds. here is the code: var queryBegin = moment(); *client.search({

Read past EOF exception on .tis and .fdt file

Hi all, we recently had an issue with ES that it reported a file corruption (more specifically a read past EOF error) after some imports/deletion for a longer timeframe. ES reported on a few nodes a long garbage collection time, but then was silent again until it started to show the EOF excepti

logstash: mutate convert doesn't work on dynamic field name

Hi, I have a csv file as follow : ,10,CPU load, ,3958,Memory used, ,...,.., So, I parse the csv. I create a new field, I use content of the 3rd column as name, and the 2nd column as value. Extract of my configuration: if [type] == "csv_perfmon" { csv { columns => [

Re: A strange behavior we've encountered on our ELK

Hi, I wrote that program and ran it and it did managed to keep a steady rate of about 1,000 events per minute even when the Kibana's total events per minute dropped from 60,000 to 6,000. However, when the Kibana's total events per minute dropped to zero, my program got a "connection refused" excep

[ANN] Elasticsearch Python language plugin 2.3.1 released

Heya, We are pleased to announce the release of the Elasticsearch Python language plugin, version 2.3.1. The Python language plugin allows to have python as the language of scripts to execute.. https://github.com/elasticsearch/elasticsearch-lang-python/ Release Notes - elasticsearch-lang-pyt

Elasticsearch completion field with jdbc river mapped data

I have made this mapping using jdbc river plugin for Elasticsearch: localhost:9200/_river/product/_meta { "type": "jdbc", "jdbc": { "url": "jdbc:sqlserver://db.example.com;databaseName=training2", "user": "root", "password": "", "sql": "SELECT uuid as uuid, caption as capti

Re: Hibernate Search with Elasticsearch

I think you need to change: https://github.com/dadoonet/legacy-search/blob/01-direct/src/main/resources/jdbc.properties https://github.com/dadoonet/legacy-search/blob/01-direct/pom.xml#L88-L92