date:20150212

David, 
 
I fixed the dependency problem - couldn't find the empty-99 commons-logging.
 
However, I'm getting this now:

11:18:46,164 WARN [JDBCExceptionReporter] SQL Error: 0, SQLState: null

11:18:46,164 ERROR [JDBCExceptionReporter] Cannot create 
PoolableConnectionFactory (Connection refused. Check that the hostname and 
port are correct and that the postmaster is accepting TCP/IP connections.)

11:18:46,164 WARN [SettingsFactory] Could not obtain connection to query 
metadata

 

Is it because I'm trying to connect to the wrong database? If so, I have 
MySQL on my computer, but I don't know how to incorporate it in the jbdc 
properties you provided me. Where do I get the driverClassName and the url??

 

 

-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d6c6dc74-ee2d-4add-9989-a21bd76c3b8b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Hibernate Search with Elasticsearch

Hi David, 
I managed to figure out the MySQL things! 
I'm now getting these errors and I don't know why. I'm testing only an 
empty database called test, there aren't any tables in it:
 

11:43:10,397 ERROR [SchemaExport] Unsuccessful: alter table Person add 
constraint FK8E4887752ACD7745 foreign key (marketing_id) references 
Marketing

11:43:10,397 ERROR [SchemaExport] Cannot add foreign key constraint

11:43:10,523 ERROR [SchemaExport] Unsuccessful: alter table Person add 
constraint FK8E488775BD8AD345 foreign key (address_id) references Address

11:43:10,523 ERROR [SchemaExport] Cannot add foreign key constraint

11:43:10,523 ERROR [SchemaExport] Unsuccessful: create sequence 
hibernate_sequence

11:43:10,523 ERROR [SchemaExport] You have an error in your SQL syntax; 
check the manual that corresponds to your MySQL server version for the 
right syntax to use near 'sequence hibernate_sequence' at line 1

-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f12b9057-e17b-494a-bfaf-e9ca7c6e8906%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Hibernate Search with Elasticsearch

I think you need to modify this line as well: 
https://github.com/dadoonet/legacy-search/blob/00-legacy/src/main/webapp/WEB-INF/person-dao-context.xml#L29
 
https://github.com/dadoonet/legacy-search/blob/00-legacy/src/main/webapp/WEB-INF/person-dao-context.xml#L29

-- 
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr 
https://twitter.com/elasticsearchfr | @scrutmydocs 
https://twitter.com/scrutmydocs



 Le 12 févr. 2015 à 20:44, William Yang williamyan...@gmail.com a écrit :
 
 Hi David,
 I managed to figure out the MySQL things!
 I'm now getting these errors and I don't know why. I'm testing only an empty 
 database called test, there aren't any tables in it:
  
 11:43:10,397 ERROR [SchemaExport] Unsuccessful: alter table Person add 
 constraint FK8E4887752ACD7745 foreign key (marketing_id) references Marketing
 
 11:43:10,397 ERROR [SchemaExport] Cannot add foreign key constraint
 
 11:43:10,523 ERROR [SchemaExport] Unsuccessful: alter table Person add 
 constraint FK8E488775BD8AD345 foreign key (address_id) references Address
 
 11:43:10,523 ERROR [SchemaExport] Cannot add foreign key constraint
 
 11:43:10,523 ERROR [SchemaExport] Unsuccessful: create sequence 
 hibernate_sequence
 
 11:43:10,523 ERROR [SchemaExport] You have an error in your SQL syntax; check 
 the manual that corresponds to your MySQL server version for the right syntax 
 to use near 'sequence hibernate_sequence' at line 1
 
 
 -- 
 You received this message because you are subscribed to the Google Groups 
 elasticsearch group.
 To unsubscribe from this group and stop receiving emails from it, send an 
 email to elasticsearch+unsubscr...@googlegroups.com 
 mailto:elasticsearch+unsubscr...@googlegroups.com.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/elasticsearch/f12b9057-e17b-494a-bfaf-e9ca7c6e8906%40googlegroups.com
  
 https://groups.google.com/d/msgid/elasticsearch/f12b9057-e17b-494a-bfaf-e9ca7c6e8906%40googlegroups.com?utm_medium=emailutm_source=footer.
 For more options, visit https://groups.google.com/d/optout 
 https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/30F9D287-F683-43B2-943A-EF5E9373EED7%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Re: Tips for failing handshake with shield

Thanks Jay for your answer. Had a good look at the appendix with
troubleshooting, but cannot really find something to help me out. On the
client I see this message:

[2015-02-12 20:10:28,363][ERROR][shield.transport.netty   ] [jc-pi-glas]
SSL/TLS handshake failed, closing channel: null

There is a stack trace after this error:
[2015-02-12 20:10:28,264][WARN ][shield.transport.netty   ] [jc-pi-glas]
exception caught on transport layer [[id: 0x127799ce, /192.168.1.11:50409
= 192.168.1.10/192.168.1.10:9300]], closing connection
java.lang.IllegalStateException: Internal error
at sun.security.ssl.SSLEngineImpl.initHandshaker(SSLEngineImpl.java:464)
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1001)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:901)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:775)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at
org.elasticsearch.common.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1285)
at
org.elasticsearch.common.netty.handler.ssl.SslHandler.decode(SslHandler.java:917)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.jav


I also get numerous timeouts. The strange thing is that these timeout are
pings to the node itself.

[2015-02-12 20:10:25,151][WARN ][discovery.zen.ping.unicast] [jc-pi-glas]
failed to send ping to
[[jc-pi-glas][RBV7_u5yRsChF_M8Ep2ICQ][jc-pi-glas][inet[
192.168.1.11/192.168.1.11:9300]]]
org.elasticsearch.transport.ReceiveTimeoutTransportException:
[jc-pi-glas][inet[
192.168.1.11/192.168.1.11:9300]][internal:discovery/zen/unicast] request_id
[5] timed out after [3751ms]
at
org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:366)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:744)


On the red pi I get this message:
[2015-02-12 20:10:44,245][WARN ][shield.transport.netty   ] [jc-pi-red]
exception caught on transport layer [[id: 0xdc4d94e1, /192.168.1.11:50409
= /192.168.1.10:9300]], closing connection
javax.net.ssl.SSLException: Received close_notify during handshake
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1646)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1614)

I also created a very small java application with sockets and used the same
jks files to authenticate. There it works.

Any other ideas?


On Wed, Feb 11, 2015 at 7:05 PM, Jay Modi jay.m...@elasticsearch.com
wrote:

 There should be another message on the other end of the connection that
 has more details of the actual failure. That message can be seen on the
 client side of the connection when the connection was closed by the server
 side.

 Also, please see
 http://www.elasticsearch.org/guide/en/shield/current/trouble-shooting.html#_sslhandshakeexception_causing_connections_to_fail
 for common problems and some tips on how to resolve them.

 On Wednesday, February 11, 2015 at 8:54:27 AM UTC-8, Jettro Coenradie
 wrote:

 Hi,
 I am trying to get SSL/TLS to work on my raspberry pi cluster. To bad I
 get the following message:

 [2015-02-11 17:51:35,037][ERROR][shield.transport.netty   ] [jc-pi-red]
 SSL/TLS handshake failed, closing channel: null

 So I guess something is wrong in my certificate chain. I have created my
 own CA and followed the steps from the getting started guide. Any tips on
 how to start debugging?

 thanks

 Jettro

  --
 You received this message because you are subscribed to a topic in the
 Google Groups elasticsearch group.
 To unsubscribe from this topic, visit
 https://groups.google.com/d/topic/elasticsearch/g-AT4CAVBCw/unsubscribe.
 To unsubscribe from this group and all its topics, send an email to
 elasticsearch+unsubscr...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/elasticsearch/652d10f6-79ee-45a5-b919-eca3bc15a5d6%40googlegroups.com
 https://groups.google.com/d/msgid/elasticsearch/652d10f6-79ee-45a5-b919-eca3bc15a5d6%40googlegroups.com?utm_medium=emailutm_source=footer
 .

 For more options, visit https://groups.google.com/d/optout.




-- 
Jettro Coenradie
http://www.gridshore.nl

-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CABB4caMUbzCHTUSn0VcQfiep24nMpYnweJ796_s4N0bbc8F%2Biw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Hibernate Search with Elasticsearch

I still need this line even though I want to replace Hibernate with 
ElasticSearch?

-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/956c6735-3221-49d4-9e3e-b6ef2c2e89d1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Hibernate Search with Elasticsearch

Well. I thought you asked me to run the example I wrote.
If so, the example uses in 00-legacy branch a database, insert, get, delete
(CRUD) and search using the database (with hibernate).
Branch 01-direct adds elasticsearch for the search part. Hibernate and the
database is still used for CRUD operations.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr
https://twitter.com/elasticsearchfr | @scrutmydocs
https://twitter.com/scrutmydocs

Le 12 févr. 2015 à 21:00, William Yang williamyan...@gmail.com a écrit :

I still need this line even though I want to replace Hibernate with
ElasticSearch?

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com
mailto:elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/956c6735-3221-49d4-9e3e-b6ef2c2e89d1%40googlegroups.com

https://groups.google.com/d/msgid/elasticsearch/956c6735-3221-49d4-9e3e-b6ef2c2e89d1%40googlegroups.com?utm_medium=emailutm_source=footer.
For more options, visit https://groups.google.com/d/optout
https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5B4195C3-6498-4DCB-A58F-74CDF9F6FF01%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Re: Hibernate Search with Elasticsearch

Hi again David,
 
The error that I get when trying to do jetty:run is:
 
 
[ERROR] Failed to execute goal on project legacy-search: Could not resolve 
dependencies for project fr.pilato.demo:legacy-search:war:1.0-SNAPSHOT: 
Could not find artifact commons-logging:commons-logging:jar:99-empty in 
central (http://repo.maven.apache.org/maven2) - [Help 1]
 
Do you know what's happening here?

-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d51844a3-ee0a-4c4d-bdc5-3da78afea105%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

logstash: mutate convert doesn't work on dynamic field name

2015-02-12 Thread Cyril SANTUNE

Hi,

I have a csv file as follow :
timestamp,10,CPU load,
timestamp,3958,Memory used,
timestamp,...,..,

So, I parse the csv. I create a new field, I use content of the 3rd column 
as name, and the 2nd column as value.

Extract of my configuration:

if [type] == csv_perfmon {
csv {
  columns = [
syslog_timestamp,
value,
name ]
  add_field = {
syslog_severity = notice
syslog_severity_code = 5
syslog_hostname = 10.68.63.110
syslog_program = serverAgent
%{name} = %{value}
  }
}
date {
  match = [ syslog_timestamp, UNIX_MS ]
  target = syslog_timestamp
}
 
mutate {
  rename = [ message, syslog_message ]
  convert = [ %{name}, float ]
  remove_field = [
value,
name,
column4,
column5
. ]
}
  }


Everything is fine in kibana/elasticsearch, I can see the new field and its 
value.
However I can not change the type of the field.
The mutate part doesn't work :

 mutate {
  convert = [ %{name}, float ]

The field type is still a string. So I can't graph it in kibana


Note: If I don't use a variable, it works
 mutate {
  convert = [ CPU load, float ]


Is it a bug or I miss something ?

-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/61a0c0d6-8748-4266-a810-5f51ce1829d6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: A strange behavior we've encountered on our ELK

Yes - can you try using the bulk API? Also, are you running on a cloud
server?

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer  Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 11:28 AM, Yuval Khalifa iyuv...@gmail.com wrote:

 Hi,

 I wrote that program and ran it and it did managed to keep a steady rate
 of about 1,000 events per minute even when the Kibana's total events per
 minute dropped from 60,000 to 6,000. However, when the
 Kibana's total events per minute dropped to zero, my program got a
 connection refused exception. I ran netstat -s and found out that every
 time the Kibana's line hit zero the number of RX-DRP increased. At that
 point I understood that I forgot to mention that this server has a 10GbE
 nic. Is it possible that the packets are being dropped because of some
 bufferis filling up? If so, how can I test it and verify that this is
 actually the case? If it is, how can I solve it?

 Thanks,
 Yuval.
 On Wednesday, February 11, 2015, Yuval Khalifa iyuv...@gmail.com wrote:

 Hi.

 When you say see how the file behaves I'm not quite sure what you mean
 by that... As I mentioned earlier, it's not that events do not appear at
 all but instead, the RATE at which they come decreases, so how can I
 measure the events rate in a file? I thought that there's another way that
 I can test this: I'll write a quick-and-dirty program that will send an
 event to the ELK via TCP every 12ms which should result in events rate of
 about 5,000 events per minute and I'll let you know if the events rate
 continues to drop or not...


 Thanks,
 Yuval.

 On Tuesday, February 10, 2015, Itamar Syn-Hershko ita...@code972.com
 wrote:

 I'd start by using logstash with input tcp and output fs and see how the
 file behaves. Same for the fs inputs - see how their files behave. And take
 it from there.

 --

 Itamar Syn-Hershko
 http://code972.com | @synhershko https://twitter.com/synhershko
 Freelance Developer  Consultant
 Lucene.NET committer and PMC member

 On Tue, Feb 10, 2015 at 7:47 PM, Yuval Khalifa iyuv...@gmail.com
 wrote:

 Great! How can I check that?


 On Tuesday, February 10, 2015, Itamar Syn-Hershko ita...@code972.com
 wrote:

 The graphic you sent suggests the issue is with logstash - since the
 @timestamp field is being populated by logstash and is the one that is 
 used
 to display the date histogram graphics in Kibana. I would start there. 
 I.e.
 maybe SecurityOnion buffers writes etc, and then to check the logstash
 shipper process stats.

 --

 Itamar Syn-Hershko
 http://code972.com | @synhershko https://twitter.com/synhershko
 Freelance Developer  Consultant
 Lucene.NET committer and PMC member

 On Tue, Feb 10, 2015 at 7:07 PM, Yuval Khalifa iyuv...@gmail.com
 wrote:

 Hi.

 Absolutely (but since that in the past I also worked at the helpdesk
 dept. I certainly understand why it is important to ask those Are you 
 sure
 it's plugged in? questions...). One of the logs is comming from
 SecurityOnion which logs (via bro-conn) all the connections so it must be
 sending data 24x7x365.

 Thanks for the quick reply,
 Yuval.

 On Tuesday, February 10, 2015, Itamar Syn-Hershko ita...@code972.com
 wrote:

 Are you sure your logs are generated linearly without bursts?

 --

 Itamar Syn-Hershko
 http://code972.com | @synhershko https://twitter.com/synhershko
 Freelance Developer  Consultant
 Lucene.NET committer and PMC member

 On Tue, Feb 10, 2015 at 6:29 PM, Yuval Khalifa iyuv...@gmail.com
 wrote:

 Hi,

 We just installed an ELK server and configured the logstash
 configuration to match the data that we send to it and until last 
 month it
 seems to be working fine but since then we see very strange behavior 
 in the
 Kibana, the event over time histogram shows the event rate at the 
 normal
 level for about a half an hour, then drops to about 20% of the normal 
 rate
 and then it continues to drop slowly for about two hours and then 
 stops and
 after a minute or two it returns to normal for the next half an hour 
 or so
 and the same behavior repeats. Needless to say that both the
 /var/log/logstash and /var/log/elasticsearch both show nothing since 
 the
 service started and by using tcpdump we can verify that events keep 
 coming
 in at the same rate all time. I attached our logstash configuration, 
 the
 /var/logstash/logstash.log, the /var/log/elasticsearch/clustername.log 
 and
 a screenshot of our Kibana with no filter applied so that you can see 
 the
 weird behavior that we see.

 Is there someone/somewhere that we can turn to to get some help on
 the subject?


 Thanks a lot,
 Yuval.

 --
 You received this message because you are subscribed to the Google
 Groups elasticsearch group.
 To unsubscribe from this group and stop receiving emails from it,
 send an email to elasticsearch+unsubscr...@googlegroups.com.
 To view this discussion on the web visit

Our approach to make custom facet's order

2015-02-12 Thread akkurug

Please see below

{
  aggs: {
category: {
  terms: {
field: category.name,
order: {
  sorting: asc
}
  },
  aggs: {
sorting: {
  avg: {
field: category.sortCode
  }
}
  }
}
  }
}

I think it's clear just from short viewing -we add special sorting field 
and then get avg metrics for using at order

It is working -but i am not sure that is the only way or the most effective

Advice please

-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/27805892-fb1c-4738-83d6-125f2e5d4212%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Search in plugin hangs

2015-02-12 Thread Mauricio Scheffer

I got this working by creating my own TransportClient instance instead of
using the injected Client.
Still, it would be nice to understand what's going on here, also locking up
the node like this seems like a pretty serious bug.

On Tuesday, February 10, 2015 at 12:27:07 PM UTC, Mauricio Scheffer wrote:

Hi, I'm writing a plugin that implements a ScoreFunction that needs to
look up some data from a separate index. It does that by having a Client
instance injected. This works perfectly in my box, but when I deploy it to
an EC2 cluster, one of the nodes simply hangs when calling the Client.
The output for /_cat/thread_pool is:

elasticsearch-cluster3.localdomain 127.0.1.1 0 0 0 0 0 0 3 18 0
elasticsearch-cluster2.localdomain 127.0.1.1 0 0 0 0 0 0 0 0 0
elasticsearch-cluster1.localdomain 127.0.1.1 0 0 0 0 0 0 0 0 0

those 3 active requests never finish, and even worse, blocks the node
entirely, it stops responding to all other search requests (which get
queued up and eventually the queue fills up and starts rejecting requests).
There is no CPU usage on that hanging node.
Obviously all the nodes are configured identically (deployed through
opsworks).

Any ideas? I guess injecting Client is not the way to go here? Any
alternatives worth trying?

Thanks,
Mauricio

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0a29cd69-bb0d-4a60-9563-28fa694f5a85%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: A strange behavior we've encountered on our ELK

Well SSD would also fix all the pains for my bank too... (-;

Are you sure it's caused by disk latency and not some sort of mis-tuned TCP
driver? I've read some blogs that recommeded to increase some of the
buffers at the sysctl.conf. Do you think so too?

On Thursday, February 12, 2015, Itamar Syn-Hershko ita...@code972.com
wrote:

 Yes, make sure the disk is local and not low latency shared one (e.g.
 SAN). Also SSD will probably fix all your pains.

 --

 Itamar Syn-Hershko
 http://code972.com | @synhershko https://twitter.com/synhershko
 Freelance Developer  Consultant
 Lucene.NET committer and PMC member

 On Thu, Feb 12, 2015 at 3:28 PM, Yuval Khalifa iyuv...@gmail.com
 javascript:_e(%7B%7D,'cvml','iyuv...@gmail.com'); wrote:

 Sort of... The ELK is running as a VM on a dedicated ESXi. Are there
 special configurations I should do in such a case?

 Thanks,
 Yuval.

 On Thursday, February 12, 2015, Itamar Syn-Hershko ita...@code972.com
 javascript:_e(%7B%7D,'cvml','ita...@code972.com'); wrote:

 Yes - can you try using the bulk API? Also, are you running on a cloud
 server?

 --

 Itamar Syn-Hershko
 http://code972.com | @synhershko https://twitter.com/synhershko
 Freelance Developer  Consultant
 Lucene.NET committer and PMC member

 On Thu, Feb 12, 2015 at 11:28 AM, Yuval Khalifa iyuv...@gmail.com
 wrote:

 Hi,

 I wrote that program and ran it and it did managed to keep a steady
 rate of about 1,000 events per minute even when the Kibana's total events
 per minute dropped from 60,000 to 6,000. However, when the
 Kibana's total events per minute dropped to zero, my program got a
 connection refused exception. I ran netstat -s and found out that every
 time the Kibana's line hit zero the number of RX-DRP increased. At that
 point I understood that I forgot to mention that this server has a 10GbE
 nic. Is it possible that the packets are being dropped because of some
 bufferis filling up? If so, how can I test it and verify that this is
 actually the case? If it is, how can I solve it?

 Thanks,
 Yuval.
 On Wednesday, February 11, 2015, Yuval Khalifa iyuv...@gmail.com
 wrote:

 Hi.

 When you say see how the file behaves I'm not quite sure what you
 mean by that... As I mentioned earlier, it's not that events do not appear
 at all but instead, the RATE at which they come decreases, so how can I
 measure the events rate in a file? I thought that there's another way that
 I can test this: I'll write a quick-and-dirty program that will send an
 event to the ELK via TCP every 12ms which should result in events rate of
 about 5,000 events per minute and I'll let you know if the events rate
 continues to drop or not...


 Thanks,
 Yuval.

 On Tuesday, February 10, 2015, Itamar Syn-Hershko ita...@code972.com
 wrote:

 I'd start by using logstash with input tcp and output fs and see how
 the file behaves. Same for the fs inputs - see how their files behave. 
 And
 take it from there.

 --

 Itamar Syn-Hershko
 http://code972.com | @synhershko https://twitter.com/synhershko
 Freelance Developer  Consultant
 Lucene.NET committer and PMC member

 On Tue, Feb 10, 2015 at 7:47 PM, Yuval Khalifa iyuv...@gmail.com
 wrote:

 Great! How can I check that?


 On Tuesday, February 10, 2015, Itamar Syn-Hershko 
 ita...@code972.com wrote:

 The graphic you sent suggests the issue is with logstash - since
 the @timestamp field is being populated by logstash and is the one 
 that is
 used to display the date histogram graphics in Kibana. I would start 
 there.
 I.e. maybe SecurityOnion buffers writes etc, and then to check the 
 logstash
 shipper process stats.

 --

 Itamar Syn-Hershko
 http://code972.com | @synhershko https://twitter.com/synhershko
 Freelance Developer  Consultant
 Lucene.NET committer and PMC member

 On Tue, Feb 10, 2015 at 7:07 PM, Yuval Khalifa iyuv...@gmail.com
 wrote:

 Hi.

 Absolutely (but since that in the past I also worked at
 the helpdesk dept. I certainly understand why it is important to ask 
 those
 Are you sure it's plugged in? questions...). One of the logs is 
 comming
 from SecurityOnion which logs (via bro-conn) all the connections so 
 it must
 be sending data 24x7x365.

 Thanks for the quick reply,
 Yuval.

 On Tuesday, February 10, 2015, Itamar Syn-Hershko 
 ita...@code972.com wrote:

 Are you sure your logs are generated linearly without bursts?

 --

 Itamar Syn-Hershko
 http://code972.com | @synhershko https://twitter.com/synhershko
 Freelance Developer  Consultant
 Lucene.NET committer and PMC member

 On Tue, Feb 10, 2015 at 6:29 PM, Yuval Khalifa iyuv...@gmail.com
  wrote:

 Hi,

 We just installed an ELK server and configured the logstash
 configuration to match the data that we send to it and until last 
 month it
 seems to be working fine but since then we see very strange 
 behavior in the
 Kibana, the event over time histogram shows the event rate at the 
 normal
 level for about a half an hour, then drops to about 20% of the 
 normal rate
 and then it continues to drop

Re: Multy-tenancy - level of service garantee

How do you guarantee a level of service provided any other way? Redundancy
and smart planning and design.

It's no different with ES.

On 13 February 2015 at 01:47, rondelvic...@gmail.com wrote:

Hi everyone,

After my precedent question
https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/US-BA4R_Qdc
regarding examples of clusters in production, I am wondering about
multy-tenancy and garantee of service in Elasticsearch :

*Multy-tenant cluster* : Is there a way to *garantee a level of service*
/ capacity planning for *each tenant* using the cluster (its *own indexes*)
?

Thanks,

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/7f99a562-ae18-447f-b227-cd145483dcf3%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/7f99a562-ae18-447f-b227-cd145483dcf3%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X96%3DWeOwcvoY39r0c9y5PPoZ86K8Ab3MrwKSYdC%3Dzc4vg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Is it possible to delete data/field without affecting the index?

You could may be use source exclude:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-source-field.html#include-exclude

Though I think it would be better to extract yourself content using Tika if you
are using Java and only send what you need to ES.

David

Le 12 févr. 2015 à 22:39, warpka...@gmail.com a écrit :

Hi,

I have some terabytes of documents (pdf, office, etc) stored in some system
outside of ES. Suppose I want to make them searchable with ES, however I will
never serve the original documents from ES, but from that other system.
Is it possible to send the documents to ES (e.g. via base64 encoded field and
the attachment type mapping), have ES index them and afterwards delete that
base64 field so that the real content of my documents is not stored in ES
(for cost reasons)?
Queries will then be served by ES but the real document is served by that
other system I have.

Regards,
Dieter
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d084274d-8d50-4fb7-8357-8d53f5177e1f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/591A15AE-09DC-4D63-B541-483D2788CD15%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

ElasticSearch search performance question

2015-02-12 Thread Jay Danielian

I know this is difficult to answer, the real answer is always It Depends 
:) But I am going to go ahead and hope I get some feedback here.

We are mainly using ES to issue terms searches against fields that are 
non-analyzed. We are using ES like a key value store, where once the match 
is found we parse the _source JSON and return our model. We are doing 
contact lookups, searching against (last_name AND (phone_number OR email)). 
We are issuing constant_score queries with term filters for the terms 
mentioned above. No aggregations, no sorting, no scripts, etc. Using 
JMeter, we were maxing out at around 500 search requests / sec. Average 
request time was taking around 7 seconds to complete. When the test would 
fire up, the ThreadPool Search Queue would spike to 1000 on each node and 
CPU would be maxed out, then once it finished everything would return to 
normal. So it appears healthy, and we wouldn't get any errors - just 
nowhere close to the performance we are looking for.

Setup details
- Index size 100GB with two different document mappings in the index. 
Roughly 500M documents
- three nodes c3.4xl instances on EC2 using pIOPS SSD EBS disks (although 
NOT RAID 0 - just one big volume)
- each server node on EC2 has 30GB RAM, 16GB on heap, rest for OS
- we have set mlockall on our instances
- 3 nodes are split into 6 shards for the main index
- Index is read only after it is loaded - we don't update the index ever, 
it is only for querying
- ES version 1.3.3 Java 1.7.0_51
- each server has 16 cores / node and 48 search threads with queue length 
of 1000

Assuming no stemming, free text queries - just term matching, how can we 
increase the throughput and decrease the response time for the ES queries? 
is 500 requests / sec at the top end?
Do we just need many more servers if we really want 3000 requests / sec ? I 
have read that scaling out is better for ES vs scaling up. But it feels 
that the current server farm should deliver better performance. 

Any help or tuning advice would be really appreciated. We have looked at 
many slideshares, blog posts from found.no, elasticseearch.org, etc - and 
can't really pinpoint a way to improve our setup. 

Thanks!

JD


-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/47b93b84-d929-4cad-becd-31581cd4c574%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

ElasticSearch search performance question

2015-02-12 Thread Mark Harwood

Term filters are useful for clauses that have a high hit rate in terms of reuse 
across queries. Given you are searching for what I imagine are unique records 
using low-frequency terms I expect the following to be true of the bitsets 
being cached by these filters:
1) they are rarely reused and therefore frequently evicted
2) are very wasteful eg have millions of bits in each set with only one bit 
having been set to 1. 

For these reasons it might be worth experimenting with term queries and not 
filters. 

-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5756599d-17d5-46f3-b60c-e21c380c1fe3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ES logs and cluster health

You'd have to ask the author of head why it doesn't align.

Does ES have permissions to write to the new log directory?

On 13 February 2015 at 05:47, caspertz christine.tumbu...@gmail.com wrote:

We have a 4 node ES cluster running version 1.4.1. We are using Ubuntu
and have it set up as a service. We have defined our location for logs as
a specific directory other than /var/log/elasticsearch which is the
default. When this starts, the process shows the value we set for the log
area but no logs are created. It is not due to disk being full.
-Des.default.path.logs=/data/moloch/logs - shows in the process list

Also have found that the /_plugin/head and the _cluster/health do not
always match up. Have see where the plugin shows green while the cluster
health shows yellow. Is this a bug or by design?

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/21337ce8-f922-42ce-b4d6-f09fe0c4266b%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/21337ce8-f922-42ce-b4d6-f09fe0c4266b%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-Kpt1ceCrazih%2B7czxTmvEztf7tw0OGZ2D87R7xQR3WQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Moving to production - Storage and servers

SAN and NAS are in the same boat, using network storage with ES is
problematic. DAS and JBOD are fine.
ES will run fine on hardware or virtualised.

It really comes down to your needs and budget.

On 13 February 2015 at 01:55, rondelvic...@gmail.com wrote:

Hi everyone,

After my first
https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/US-BA4R_Qdc
and second question
https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/FGKUmzn-WSs
regarding clusters' examples and multy-tenancy with garantee of service, I
have one more on my way to production :

Could you describe the *pros* and *cons* of the :

1. Disks : *DAS* (Direct Attached Storage) - *JBOD* (Just a Bunch Of
Disks) / *SAN* (Storage Area Network)? I take it that NAS is most
of the time to be avoided
2. Servers : bare metal servers / virtual servers ?

For those interested :

- JBOD / SAN storage discussion in To Raid or not to Raid:
https://groups.google.com/forum/?fromgroups#!searchin/
elasticsearch/hardware/elasticsearch/HSj2fZGdU1Y/4mFCBTCb-JcJ

https://groups.google.com/forum/?fromgroups#%21searchin/elasticsearch/hardware/elasticsearch/HSj2fZGdU1Y/4mFCBTCb-JcJ
- Doc which states about SSDs, RAID0, NAS, networks

http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/hardware.html

Thanks!

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3ab36482-e4fb-4dd0-99cd-044d2e4257ea%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/3ab36482-e4fb-4dd0-99cd-044d2e4257ea%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8Xe1CXVT3TcdWPmzd1HmE_P8VdTzB0%2BqG%3Dyj9ePSNKiw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Master Node vs. Data Node Architecture

Except that is overkill when you only have 3 nodes.

How much data do you have in the cluster?

On 13 February 2015 at 01:15, Itamar Syn-Hershko ita...@code972.com wrote:

See this:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-node.html

Basically, the recommended pattern talks about isolating responsibilities.
A node should either be a data node, master-eligible node, or an external
gateway to the cluster (client node)

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 4:08 PM, Eric eric.luel...@gmail.com wrote:

Hello,

Currently I have a 3 node ElasticSearch cluster. Each node is a RHEL VM
with 16 gig RAM. The basic config is:

- All nodes can be master and are data nodes.
- 3 shards and 1 replica
- 6 different indexes

I'm starting to run into issues of ElasticSearch bogging down on searches
and is completely freezing sometimes at night. I've dedicated 9 gig to heap
size and it says i'm using ~60% of the heap RAM and about 70% of the
overall heap. So even though I'm using quite a bit of the heap, I'm not
maxed out. I've attached a screenshot of the exact stats from Elastic HQ.
I'm averaging around 10,000 events/sec coming into the cluster from 6
different Logstash instances on another server.

My question is what can I do to help the stability and speed of my
cluster. Currently I'm having issues with 1 node going down and it taking
everything else down. The HA portion isn't working very well. I'm debating
about either adding 1 more node with the exact same stats or adding 2 more
smaller VMs that will act as master nodes only. I didn't know which one was
recommended or where I would get the biggest bang for the buck.

Any information would be greatly appreciated.

Thanks,
Eric

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/354a2326-5532-4239-87ea-f02af64fe71f%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/354a2326-5532-4239-87ea-f02af64fe71f%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZureLROJMaO7gVprFjA2OmRZA0ZYyH1v%2Bges06u_V__6w%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZureLROJMaO7gVprFjA2OmRZA0ZYyH1v%2Bges06u_V__6w%40mail.gmail.com?utm_medium=emailutm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X93EwqeGf9S4UpMvtJy3%2BmaAjovfVicj7LRHz%2BPyAbSug%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Updating several indices at once

You can use wildcards, you can also comma separate them eg - curl -XPUT
'localhost:9200/logstash-2014.01.01,logstash-2014.01.02/_settings

On 12 February 2015 at 23:53, Aldian aldian...@gmail.com wrote:

Today I tried the following request, which worked great:
curl -XPUT 'localhost:9200/logstash-2014.03.24/_settings' -d '
{
index : {
number_of_replicas : 0
}
}'

Now I want to set number of replicas for every indice in 2014 to 0. Can I
use a wildcard in the url such as the following? :
curl -XPUT 'localhost:9200/logstash-2014.*/_settings' -d '
{
index : {
number_of_replicas : 0
}
}'

Or do I have to make a shell script that will iterate over all the dates?

Thanks for your help

Aldian

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/174155d7-6133-40f0-9d8f-c340971f1f44%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/174155d7-6133-40f0-9d8f-c340971f1f44%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_2PH07k%2BjkU4arAC6FxNRQJxZM9FQpw97u_ob5JOpxvQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Is it possible to delete data/field without affecting the index?

2015-02-12 Thread warpkanal

Hi,

I have some terabytes of documents (pdf, office, etc) stored in some system 
outside of ES. Suppose I want to make them *searchable* with ES, however I 
will never serve the original documents from ES, but from that other system.
Is it possible to send the documents to ES (e.g. via base64 encoded field 
and the attachment type mapping), have ES index them and afterwards delete 
that base64 field so that the real content of my documents is not stored 
in ES (for cost reasons)?
Queries will then be served by ES but the real document is served by that 
other system I have.

Regards,
Dieter

-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d084274d-8d50-4fb7-8357-8d53f5177e1f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Is it possible to delete data/field without affecting the index?

2015-02-12 Thread warpkanal

Thanks a lot, that sounds exactly like what I was looking for!
Why would you suggest extracting the content myself? Because of the
experimental state of the attachment type plugin?
Even if I'd extract the content myself I wouldn't want to store it in ES
(as I'd never request it from ES). The only benefit I could think of is the
ability to reindex inside ES without having my outer system to feed the
content in again for reindexing.

On Thursday, February 12, 2015 at 11:02:51 PM UTC+1, David Pilato wrote:

You could may be use source exclude:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-source-field.html#include-exclude

Though I think it would be better to extract yourself content using Tika
if you are using Java and only send what you need to ES.

David

Le 12 févr. 2015 à 22:39, warp...@gmail.com javascript: a écrit :

Hi,

I have some terabytes of documents (pdf, office, etc) stored in some
system outside of ES. Suppose I want to make them *searchable* with ES,
however I will never serve the original documents from ES, but from that
other system.
Is it possible to send the documents to ES (e.g. via base64 encoded field
and the attachment type mapping), have ES index them and afterwards delete
that base64 field so that the real content of my documents is not stored
in ES (for cost reasons)?
Queries will then be served by ES but the real document is served by that
other system I have.

Regards,
Dieter

https://groups.google.com/d/msgid/elasticsearch/d084274d-8d50-4fb7-8357-8d53f5177e1f%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b37643b0-c05e-4159-86b4-3f31f8fbfb9d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Hibernate Search with Elasticsearch

Ah I see, I will try to go through the other branches as well.
 
However, when I try to do the CRUD operations (What does that stand for by 
the way?), I get a 

16:20:55,055 DEBUG [PersonRestAPI] create(1, )

16:20:55,055 DEBUG [PersonService] get(1)=null

 

Do you know the cause of that?

-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e01521fd-fee5-4e21-b0c9-05cd50892fb5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: function score filter function doesn't seem to be cached

2015-02-12 Thread Jason Wee

set explain to true and read the explain output if cache is use?

Jason

On Fri, Feb 13, 2015 at 3:47 AM, ffors...@homeaway.com wrote:

 I am using a function score with a function filter to score a subset of
 results higher. I specify to cache the function filter, but its performance
 is slower and slower the more clauses I add to the filter.

 A query with 2 geo_polygons clauses returns relatively fast (30ms), but
 the more polygons clauses I add the slower it gets. With 50 polygons the
 call slows to 1s. Does elasticsearch actually cache the filter?

 Example query with 2 geo_polygons clauses:
 {
   query: {
 function_score: {
   filter: {
 bool: {
   must: [
 {
   term: {
 city: LA
   }
 }
   ]
 }
   },
   functions: [
 {
   filter: {
 bool: {
   should: [
 {
   geo_polygon: {
 location: {
   points: [
 [
   -118.3971081,
   34.081519
 ],
 [
   -118.3440351,
   34.081519
 ],
 [
   -118.3440351,
   34.131086
 ],
 [
   -118.3971081,
   34.131086
 ]
   ]
 }
   }
 },
 {
   geo_polygon: {
 location: {
   points: [
 [
   -118.3907139,
   34.05576
 ],
 [
   -118.361375,
   34.05576
 ],
 [
   -118.361375,
   34.083811
 ],
 [
   -118.3907139,
   34.083811
 ]
   ]
 }
   }
 }
   ],
   _cache: true
 }
   },
   boost_factor: 2
 }
   ],
   score_mode: sum
 }
   }
 }

 --
 You received this message because you are subscribed to the Google Groups
 elasticsearch group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to elasticsearch+unsubscr...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/elasticsearch/c990d908-2b15-4797-9300-22a71aeb4972%40googlegroups.com
 https://groups.google.com/d/msgid/elasticsearch/c990d908-2b15-4797-9300-22a71aeb4972%40googlegroups.com?utm_medium=emailutm_source=footer
 .
 For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAHO4itze5wyDX%2Bp3OC%2BPpN%3Db1XHLQQvKb68ZEeOnAA9tEAwiBw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Tips for failing handshake with shield

2015-02-12 Thread Jay Modi

I think you are running into the TLS handshake taking too long and it 
causes the pings to timeout and then the Netty channels are closed, which 
is why you see the connection closed messages. Since you are running on a 
Raspberry PI, the overhead of TLS is going to be a lot for that platform; 
the handshake is expensive in terms of CPU. You probably need to increase 
the zen ping timeout and based on your other post it may need to be like 
over 25 seconds.

On Thursday, February 12, 2015 at 11:20:35 AM UTC-8, Jettro Coenradie wrote:

 Thanks Jay for your answer. Had a good look at the appendix with 
 troubleshooting, but cannot really find something to help me out. On the 
 client I see this message:

 [2015-02-12 20:10:28,363][ERROR][shield.transport.netty   ] [jc-pi-glas] 
 SSL/TLS handshake failed, closing channel: null

 There is a stack trace after this error:
 [2015-02-12 20:10:28,264][WARN ][shield.transport.netty   ] [jc-pi-glas] 
 exception caught on transport layer [[id: 0x127799ce, /192.168.1.11:50409 
 = 192.168.1.10/192.168.1.10:9300] 
 http://192.168.1.10/192.168.1.10:9300%5D], closing connection
 java.lang.IllegalStateException: Internal error
 at sun.security.ssl.SSLEngineImpl.initHandshaker(SSLEngineImpl.java:464)
 at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1001)
 at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:901)
 at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:775)
 at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
 at 
 org.elasticsearch.common.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1285)
 at 
 org.elasticsearch.common.netty.handler.ssl.SslHandler.decode(SslHandler.java:917)
 at 
 org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.jav


 I also get numerous timeouts. The strange thing is that these timeout are 
 pings to the node itself.

 [2015-02-12 20:10:25,151][WARN ][discovery.zen.ping.unicast] [jc-pi-glas] 
 failed to send ping to 
 [[jc-pi-glas][RBV7_u5yRsChF_M8Ep2ICQ][jc-pi-glas][inet[
 192.168.1.11/192.168.1.11:9300] http://192.168.1.11/192.168.1.11:9300%5D
 ]]
 org.elasticsearch.transport.ReceiveTimeoutTransportException: 
 [jc-pi-glas][inet[
 192.168.1.11/192.168.1.11:9300]][internal:discovery/zen/unicast 
 http://192.168.1.11/192.168.1.11:9300%5D%5D%5Binternal:discovery/zen/unicast]
  
 request_id [5] timed out after [3751ms]
 at 
 org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:366)
 at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:744)


 On the red pi I get this message:
 [2015-02-12 20:10:44,245][WARN ][shield.transport.netty   ] [jc-pi-red] 
 exception caught on transport layer [[id: 0xdc4d94e1, /192.168.1.11:50409 
 = /192.168.1.10:9300]], closing connection
 javax.net.ssl.SSLException: Received close_notify during handshake
 at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
 at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1646)
 at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1614)

 I also created a very small java application with sockets and used the 
 same jks files to authenticate. There it works.

 Any other ideas?


 On Wed, Feb 11, 2015 at 7:05 PM, Jay Modi jay@elasticsearch.com 
 javascript: wrote:

 There should be another message on the other end of the connection that 
 has more details of the actual failure. That message can be seen on the 
 client side of the connection when the connection was closed by the server 
 side.

 Also, please see 
 http://www.elasticsearch.org/guide/en/shield/current/trouble-shooting.html#_sslhandshakeexception_causing_connections_to_fail
  
 for common problems and some tips on how to resolve them.

 On Wednesday, February 11, 2015 at 8:54:27 AM UTC-8, Jettro Coenradie 
 wrote:

 Hi,
 I am trying to get SSL/TLS to work on my raspberry pi cluster. To bad I 
 get the following message:

 [2015-02-11 17:51:35,037][ERROR][shield.transport.netty   ] [jc-pi-red] 
 SSL/TLS handshake failed, closing channel: null

 So I guess something is wrong in my certificate chain. I have created my 
 own CA and followed the steps from the getting started guide. Any tips on 
 how to start debugging?

 thanks

 Jettro

  -- 
 You received this message because you are subscribed to a topic in the 
 Google Groups elasticsearch group.
 To unsubscribe from this topic, visit 
 https://groups.google.com/d/topic/elasticsearch/g-AT4CAVBCw/unsubscribe.
 To unsubscribe from this group and all its topics, send an email to 
 elasticsearc...@googlegroups.com javascript:.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/elasticsearch/652d10f6-79ee-45a5-b919-eca3bc15a5d6%40googlegroups.com

Re: ElasticSearch search performance question

2015-02-12 Thread Nikolas Everett

You might want to try hitting hot threads while putting your load on it and
seeing what you see. Or posting it.

Nik

On Thu, Feb 12, 2015 at 4:44 PM, Jay Danielian jay.daniel...@circleback.com
wrote:

Mark,

Thanks for the initial reply. Yes, your assumption about these things
being very specific and thus not likely to have any re-use with regards to
caching is correct. I have attached some screenshots from the BigDesk
plugin which showed a decent snapshot of what the server looked like while
my tests were running. You can see the spikes in CPU, that essentially
covered the duration when the JMeter tests were running.

At a high level, the only thing that seems to be really stressed on the
server is CPU. But that makes me think that there is something in my setup
, query syntax, or perhaps the cache eviction rate, etc that is causing it
to spike so high. I also have concerns about non RAID 0 the EBS volumes, as
I know that having one large volume does not maximize throughput - however,
just looking at the stats it doesn't seem like IO is really a bottleneck.

Here is a sample query structure =
https://gist.github.com/jaydanielian/c2be885987f344031cfc

Also this is one query - in reality we use _msearch to pipeline several of
these queries in one batch. The queries also include custom routing / route
key to make sure we only hit one shard.

Thanks!

On Thursday, February 12, 2015 at 4:22:29 PM UTC-5, Mark Walkom wrote:

It'd help if you could gist/pastebin/etc a query example.

Also your current ES and java need updating, there are known issues with
java 1.7u55, and you will always see performance boosts running the latest
version of ES.

That aside, what is your current resource utilisation like? Are you
seeing lots of cache evictions, high heap use, high CPU, IO delays?

On 13 February 2015 at 07:32, Jay Danielian jay.da...@circleback.com
wrote:

I know this is difficult to answer, the real answer is always It
Depends :) But I am going to go ahead and hope I get some feedback here.

We are mainly using ES to issue terms searches against fields that are
non-analyzed. We are using ES like a key value store, where once the match
is found we parse the _source JSON and return our model. We are doing
contact lookups, searching against (last_name AND (phone_number OR email)).
We are issuing constant_score queries with term filters for the terms
mentioned above. No aggregations, no sorting, no scripts, etc. Using
JMeter, we were maxing out at around 500 search requests / sec. Average
request time was taking around 7 seconds to complete. When the test would
fire up, the ThreadPool Search Queue would spike to 1000 on each node and
CPU would be maxed out, then once it finished everything would return to
normal. So it appears healthy, and we wouldn't get any errors - just
nowhere close to the performance we are looking for.

Setup details
- Index size 100GB with two different document mappings in the index.
Roughly 500M documents
- three nodes c3.4xl instances on EC2 using pIOPS SSD EBS disks
(although NOT RAID 0 - just one big volume)
- each server node on EC2 has 30GB RAM, 16GB on heap, rest for OS
- we have set mlockall on our instances
- 3 nodes are split into 6 shards for the main index
- Index is read only after it is loaded - we don't update the index
ever, it is only for querying
- ES version 1.3.3 Java 1.7.0_51
- each server has 16 cores / node and 48 search threads with queue
length of 1000

Assuming no stemming, free text queries - just term matching, how can we
increase the throughput and decrease the response time for the ES queries?
is 500 requests / sec at the top end?
Do we just need many more servers if we really want 3000 requests / sec
? I have read that scaling out is better for ES vs scaling up. But it feels
that the current server farm should deliver better performance.

Any help or tuning advice would be really appreciated. We have looked at
many slideshares, blog posts from found.no, elasticseearch.org, etc -
and can't really pinpoint a way to improve our setup.

Thanks!

--
You received this message because you are subscribed to the Google
Groups elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/47b93b84-d929-4cad-becd-31581cd4c574%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/47b93b84-d929-4cad-becd-31581cd4c574%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

Re: Master Node vs. Data Node Architecture

Depending why the node goes down - going mid-way with dedicated master
nodes is sometimes the solution

And if this is due to massive use of aggregations, doc-values may be the
answer (or larger heap, but that's costlier)

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 11:40 PM, Mark Walkom markwal...@gmail.com wrote:

Except that is overkill when you only have 3 nodes.

How much data do you have in the cluster?

On 13 February 2015 at 01:15, Itamar Syn-Hershko ita...@code972.com
wrote:

See this:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-node.html

Basically, the recommended pattern talks about isolating
responsibilities. A node should either be a data node, master-eligible
node, or an external gateway to the cluster (client node)

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 4:08 PM, Eric eric.luel...@gmail.com wrote:

Hello,

Currently I have a 3 node ElasticSearch cluster. Each node is a RHEL VM
with 16 gig RAM. The basic config is:

- All nodes can be master and are data nodes.
- 3 shards and 1 replica
- 6 different indexes

I'm starting to run into issues of ElasticSearch bogging down on
searches and is completely freezing sometimes at night. I've dedicated 9
gig to heap size and it says i'm using ~60% of the heap RAM and about 70%
of the overall heap. So even though I'm using quite a bit of the heap, I'm
not maxed out. I've attached a screenshot of the exact stats from Elastic
HQ. I'm averaging around 10,000 events/sec coming into the cluster from 6
different Logstash instances on another server.

Any information would be greatly appreciated.

Thanks,
Eric

--
You received this message because you are subscribed to the Google
Groups elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/354a2326-5532-4239-87ea-f02af64fe71f%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/354a2326-5532-4239-87ea-f02af64fe71f%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZureLROJMaO7gVprFjA2OmRZA0ZYyH1v%2Bges06u_V__6w%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZureLROJMaO7gVprFjA2OmRZA0ZYyH1v%2Bges06u_V__6w%40mail.gmail.com?utm_medium=emailutm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X93EwqeGf9S4UpMvtJy3%2BmaAjovfVicj7LRHz%2BPyAbSug%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X93EwqeGf9S4UpMvtJy3%2BmaAjovfVicj7LRHz%2BPyAbSug%40mail.gmail.com?utm_medium=emailutm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZvFX-bEqpSEnh3EmdbzAwAhDLE7PYDivd5Q2VnFu_xviA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Is Unsecured OPTIONS Method A Vulnerability?

A quick check shows that ES returns nothing with an options request via
curl.

ES uses netty to serve HTTP content.

On 12 February 2015 at 08:50, stv.bel...@gmail.com wrote:

I've been working lately on a project utilizing ElasticSearch and Kibana.
To secure the ElasticSearch API I've hidden it behind a reverse proxy.

The proxy uses a cookie to authenticate the request and forward it to the
ElasticSearch server, but if no cookie is present or if the cookie does not
validate, then 401 is returned.

Here's the catch. Kibana uses CORS to communicate with ElasticSearch, so
while I can enable the Kibana HTTP client to use the withCredentials option
which will include cookies, it only does so for the four CRUD HTTP verbs.
Glaringly, any OPTIONS requests from Kibana will not include the cookie.

This makes sense on a certain level due to the description of the intended
purpose for the OPTIONS verb in the HTTP spec.

As such, in order to get my front-end functioning through this reverse
proxy I've had to white-list all OPTIONS requests. I'm concerned with
whether or not this could be abused to get commands through to the ES
server that I otherwise wouldn't want. I trust that Kibana is using the
verb properly, but if an attacker crafted an OPTIONS request at a server
with the request /_shutdown, would the ElasticServer know that since this
is an OPTIONS request it should ignore anything else in the request?

Admittedly I'm a bit in the dark about how the ES server receives and
handles commands over http beyond the typical RESTful functionality. Anyone
can shed some light?

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3be24cfc-c247-4ab7-9733-e494f527529b%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/3be24cfc-c247-4ab7-9733-e494f527529b%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X91QFbU%3DFXWXqOBS1rv7BH98YKBnnoWMB69sSgM_L0Rqg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: A strange behavior we've encountered on our ELK

Hi,

I wrote that program and ran it and it did managed to keep a steady rate of
about 1,000 events per minute even when the Kibana's total events per
minute dropped from 60,000 to 6,000. However, when the
Kibana's total events per minute dropped to zero, my program got a
connection refused exception. I ran netstat -s and found out that every
time the Kibana's line hit zero the number of RX-DRP increased. At that
point I understood that I forgot to mention that this server has a 10GbE
nic. Is it possible that the packets are being dropped because of some
bufferis filling up? If so, how can I test it and verify that this is
actually the case? If it is, how can I solve it?

Thanks,
Yuval.
On Wednesday, February 11, 2015, Yuval Khalifa iyuv...@gmail.com wrote:

Hi.

When you say see how the file behaves I'm not quite sure what you mean
by that... As I mentioned earlier, it's not that events do not appear at
all but instead, the RATE at which they come decreases, so how can I
measure the events rate in a file? I thought that there's another way that
I can test this: I'll write a quick-and-dirty program that will send an
event to the ELK via TCP every 12ms which should result in events rate of
about 5,000 events per minute and I'll let you know if the events rate
continues to drop or not...

Thanks,
Yuval.

On Tuesday, February 10, 2015, Itamar Syn-Hershko ita...@code972.com
javascript:_e(%7B%7D,'cvml','ita...@code972.com'); wrote:

I'd start by using logstash with input tcp and output fs and see how the
file behaves. Same for the fs inputs - see how their files behave. And take
it from there.

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer Consultant
Lucene.NET committer and PMC member

On Tue, Feb 10, 2015 at 7:47 PM, Yuval Khalifa iyuv...@gmail.com wrote:

Great! How can I check that?

On Tuesday, February 10, 2015, Itamar Syn-Hershko ita...@code972.com
wrote:

The graphic you sent suggests the issue is with logstash - since the
@timestamp field is being populated by logstash and is the one that is used
to display the date histogram graphics in Kibana. I would start there. I.e.
maybe SecurityOnion buffers writes etc, and then to check the logstash
shipper process stats.

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer Consultant
Lucene.NET committer and PMC member

On Tue, Feb 10, 2015 at 7:07 PM, Yuval Khalifa iyuv...@gmail.com
wrote:

Hi.

Absolutely (but since that in the past I also worked at the helpdesk
dept. I certainly understand why it is important to ask those Are you
sure
it's plugged in? questions...). One of the logs is comming from
SecurityOnion which logs (via bro-conn) all the connections so it must be
sending data 24x7x365.

Thanks for the quick reply,
Yuval.

On Tuesday, February 10, 2015, Itamar Syn-Hershko ita...@code972.com
wrote:

Are you sure your logs are generated linearly without bursts?

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer Consultant
Lucene.NET committer and PMC member

On Tue, Feb 10, 2015 at 6:29 PM, Yuval Khalifa iyuv...@gmail.com
wrote:

Hi,

We just installed an ELK server and configured the logstash
configuration to match the data that we send to it and until last month
it
seems to be working fine but since then we see very strange behavior in
the
Kibana, the event over time histogram shows the event rate at the normal
level for about a half an hour, then drops to about 20% of the normal
rate
and then it continues to drop slowly for about two hours and then stops
and
after a minute or two it returns to normal for the next half an hour or
so
and the same behavior repeats. Needless to say that both the
/var/log/logstash and /var/log/elasticsearch both show nothing since the
service started and by using tcpdump we can verify that events keep
coming
in at the same rate all time. I attached our logstash configuration, the
/var/logstash/logstash.log, the /var/log/elasticsearch/clustername.log
and
a screenshot of our Kibana with no filter applied so that you can see
the
weird behavior that we see.

Is there someone/somewhere that we can turn to to get some help on
the subject?

Thanks a lot,
Yuval.

--
You received this message because you are subscribed to the Google
Groups elasticsearch group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this

node.js client timeout doesn't work (for me)

2015-02-12 Thread Lior Goldemberg

hi guys,

i wonder if someone had the same problem.
i have a heavy query, with complicated script, and i want to give this 
query larger timeout.

but no matter, what number i put there, the timeout happens after 30 
seconds.

here is the code:

var queryBegin = moment();










*client.search({index: strIndices,type: xxx,ignoreUnavailable: 
true,timeout: 60,body: body}, function (error, response) {var 
queryEnd = moment();console.log(query callback. duration:  
+(queryEnd-queryBegin) + ** ms **);*




and the console shows:

*getDroppingConversion callback. duration: 30009 ms
Request Timeout after 3ms*


thanks a lot

Lior


-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/cdd23d99-9e1d-40b0-bfd5-7ba5657bc0c9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

mysql to elasticsearch replication using elasticsearch-river-jdbc

2015-02-12 Thread Mahesh Sankaran

Hi all,
   I am working on mysql to ES replication using 
elasticsearch-river-jdbc.my configuration is,

curl -XPUT '10.10.10.53:9200/_river/test/_meta' -d '{
type : jdbc,
  jdbc : {
  driver : com.mysql.jdbc.Driver,
  url : jdbc:mysql://localhost:3306/Mahesh,
  user : root,
  password : root,
   sql : select id as _id,name,age  from main ,
   schedule: * * * * * ? ,
   autocommit : true,
   strategy : simple

} }'

It replicates data every time when mysql table got updated.mysql table has 
nine number of rows.when i see the elasticsearch logs, every time the 
number of rows get doubled.


2015-02-13 10:24:28,139][INFO ][river.jdbc.RiverMetrics  ] pipeline 
org.xbib.elasticsearch.plugin.jdbc.RiverPipeline@204426cd complete: river 
jdbc/test metrics: 9 rows, 3.0049217412872427 mean, (0.0 0.0 0.0), ingest 
metrics: elapsed 2 seconds, 643.0 bytes bytes, 64.0 bytes avg, 0 MB/s
[2015-02-13 10:24:28,160][INFO ][index.shard.service  ] [Volla] 
[jdbc][0] updating refresh_interval from [1s] to [-1]
[2015-02-13 10:24:28,160][INFO ][index.shard.service  ] [Volla] 
[jdbc][3] updating refresh_interval from [1s] to [-1]
[2015-02-13 10:24:28,160][INFO ][index.shard.service  ] [Volla] 
[jdbc][1] updating refresh_interval from [1s] to [-1]
[2015-02-13 10:24:28,160][INFO ][index.shard.service  ] [Volla] 
[jdbc][2] updating refresh_interval from [1s] to [-1]
[2015-02-13 10:24:28,161][INFO ][index.shard.service  ] [Volla] 
[jdbc][4] updating refresh_interval from [1s] to [-1]
[2015-02-13 10:24:28,257][INFO ][index.shard.service  ] [Volla] 
[jdbc][0] updating refresh_interval from [-1] to [1s]
[2015-02-13 10:24:28,258][INFO ][index.shard.service  ] [Volla] 
[jdbc][3] updating refresh_interval from [-1] to [1s]
[2015-02-13 10:24:28,258][INFO ][index.shard.service  ] [Volla] 
[jdbc][1] updating refresh_interval from [-1] to [1s]
[2015-02-13 10:24:28,258][INFO ][index.shard.service  ] [Volla] 
[jdbc][2] updating refresh_interval from [-1] to [1s]
[2015-02-13 10:24:28,258][INFO ][index.shard.service  ] [Volla] 
[jdbc][4] updating refresh_interval from [-1] to [1s]
[2015-02-13 10:24:28,387][INFO ][river.jdbc.RiverMetrics  ] pipeline 
org.xbib.elasticsearch.plugin.jdbc.RiverPipeline@7dcf6e34 complete: river 
jdbc/test metrics: 18 rows, 5.544464111788077 mean, (0.0 0.0 0.0), ingest 
metrics: elapsed 0 seconds, 643.0 bytes bytes, 33.0 bytes avg, 0.003 MB/s
[2015-02-13 10:24:28,410][INFO ][index.shard.service  ] [Volla] 
[jdbc][0] updating refresh_interval from [1s] to [-1]
[2015-02-13 10:24:28,417][INFO ][index.shard.service  ] [Volla] 
[jdbc][3] updating refresh_interval from [1s] to [-1]
[2015-02-13 10:24:28,417][INFO ][index.shard.service  ] [Volla] 
[jdbc][1] updating refresh_interval from [1s] to [-1]
[2015-02-13 10:24:28,417][INFO ][index.shard.service  ] [Volla] 
[jdbc][2] updating refresh_interval from [1s] to [-1]
[2015-02-13 10:24:28,417][INFO ][index.shard.service  ] [Volla] 
[jdbc][4] updating refresh_interval from [1s] to [-1]
[2015-02-13 10:24:28,509][INFO ][index.shard.service  ] [Volla] 
[jdbc][0] updating refresh_interval from [-1] to [1s]
[2015-02-13 10:24:28,509][INFO ][index.shard.service  ] [Volla] 
[jdbc][3] updating refresh_interval from [-1] to [1s]
[2015-02-13 10:24:28,509][INFO ][index.shard.service  ] [Volla] 
[jdbc][1] updating refresh_interval from [-1] to [1s]
[2015-02-13 10:24:28,510][INFO ][index.shard.service  ] [Volla] 
[jdbc][2] updating refresh_interval from [-1] to [1s]
[2015-02-13 10:24:28,510][INFO ][index.shard.service  ] [Volla] 
[jdbc][4] updating refresh_interval from [-1] to [1s]
[2015-02-13 10:24:28,619][INFO ][river.jdbc.RiverMetrics  ] pipeline 
org.xbib.elasticsearch.plugin.jdbc.RiverPipeline@18630651 complete: river 
jdbc/test metrics: 27 rows, 7.762557850074249 mean, (0.0 0.0 0.0), ingest 
metrics: elapsed 0 seconds, 643.0 bytes bytes, 22.0 bytes avg, 0.003 MB/s
[2015-02-13 10:24:29,011][INFO ][index.shard.service  ] [Volla] 
[jdbc][0] updating refresh_interval from [1s] to [-1]
[2015-02-13 10:24:29,011][INFO ][index.shard.service  ] [Volla] 
[jdbc][3] updating refresh_interval from [1s] to [-1]
[2015-02-13 10:24:29,011][INFO ][index.shard.service  ] [Volla] 
[jdbc][1] updating refresh_interval from [1s] to [-1]
[2015-02-13 10:24:29,011][INFO ][index.shard.service  ] [Volla] 
[jdbc][2] updating refresh_interval from [1s] to [-1]
[2015-02-13 10:24:29,012][INFO ][index.shard.service  ] [Volla] 
[jdbc][4] updating refresh_interval from [1s] to [-1]
[2015-02-13 10:24:29,083][INFO ][index.shard.service  ] [Volla] 
[jdbc][0] updating refresh_interval from [-1] to [1s]
[2015-02-13 10:24:29,083][INFO ][index.shard.service  ] [Volla] 
[jdbc][3] updating refresh_interval from [-1] to [1s]
[2015-02-13 10:24:29,083][INFO ][index.shard.service  ] [Volla] 
[jdbc][1] updating refresh_interval from [-1] to [1s]
[2015-02-13 10:24:29,084][INFO ][index.shard.service  ] [Volla]

Re: Hibernate Search with Elasticsearch

CRUD=Create, Read, Update, Delete

No. What exact operation did you run?

David

 Le 13 févr. 2015 à 01:22, William Yang williamyan...@gmail.com a écrit :
 
 Ah I see, I will try to go through the other branches as well.
  
 However, when I try to do the CRUD operations (What does that stand for by 
 the way?), I get a
 16:20:55,055 DEBUG [PersonRestAPI] create(1, )
 
 16:20:55,055 DEBUG [PersonService] get(1)=null
 
  
 
 Do you know the cause of that?
 
 -- 
 You received this message because you are subscribed to the Google Groups 
 elasticsearch group.
 To unsubscribe from this group and stop receiving emails from it, send an 
 email to elasticsearch+unsubscr...@googlegroups.com.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/elasticsearch/e01521fd-fee5-4e21-b0c9-05cd50892fb5%40googlegroups.com.
 For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4B3E165A-DC75-4462-BD46-A64739FFE710%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Kibana authentication proxy?

2015-02-12 Thread Siddharth Trikha

I want authentication for my elastic-search cluster so that security is 
handled. So I came across Kibana authentication proxy 
https://github.com/fangli/kibana-authentication-proxy


So I installed it and got the app.js running.

But there are no instructions on how to use it to achieve authentication.

Or if there is a better solution to authentication, then please suggest 
that.
Any help is appreciated.


-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b172cc03-f31e-45bd-a6b2-c9fd426b8331%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Shield: Why do I get timeout on my own node

Hi I am trying to create a cluster that is secured on a number of raspberry 
pi's. If I start just one node with the following config I get a timeout to 
that same node. So it is trying to ping itself and that does not work. Any 
ideas?

Config:
cluster.name: jc-mypi
node.name: jc-pi-red
marvel.agent.indices: my*
marvel.agent.interval: 10s
marvel.agent.enabled: false
shield.audit.enabled: true
shield.ssl.keystore.path: /etc/elasticsearch/jcpi.jks
shield.ssl.keystore.password: geheim
shield.transport.ssl: true
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: [192.168.1.11:9300, 192.168.1.12:9300]

In the log:

[2015-02-12 21:36:40,078][WARN ][shield.transport ] [jc-pi-red] 
Received response for a request that has timed out, sent [21455ms] ago, 
timed out [17704ms] ago, action [internal:discovery/zen/unicast], node 
[[jc-pi-red][o4b8he9qQFiSzBV3ckkF1Q][jc-pi-red][inet[192.168.1.10/192.168.1.10:9300]]],
 
id [0]

Thanks

-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/791792bc-9ee3-48ca-9f61-e2bd71952cc9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ElasticSearch search performance question

It'd help if you could gist/pastebin/etc a query example.

Also your current ES and java need updating, there are known issues with
java 1.7u55, and you will always see performance boosts running the latest
version of ES.

That aside, what is your current resource utilisation like? Are you seeing
lots of cache evictions, high heap use, high CPU, IO delays?

On 13 February 2015 at 07:32, Jay Danielian jay.daniel...@circleback.com
wrote:

I know this is difficult to answer, the real answer is always It Depends
:) But I am going to go ahead and hope I get some feedback here.

Setup details
- Index size 100GB with two different document mappings in the index.
Roughly 500M documents
- three nodes c3.4xl instances on EC2 using pIOPS SSD EBS disks (although
NOT RAID 0 - just one big volume)
- each server node on EC2 has 30GB RAM, 16GB on heap, rest for OS
- we have set mlockall on our instances
- 3 nodes are split into 6 shards for the main index
- Index is read only after it is loaded - we don't update the index ever,
it is only for querying
- ES version 1.3.3 Java 1.7.0_51
- each server has 16 cores / node and 48 search threads with queue length
of 1000

Assuming no stemming, free text queries - just term matching, how can we
increase the throughput and decrease the response time for the ES queries?
is 500 requests / sec at the top end?
Do we just need many more servers if we really want 3000 requests / sec ?
I have read that scaling out is better for ES vs scaling up. But it feels
that the current server farm should deliver better performance.

Any help or tuning advice would be really appreciated. We have looked at
many slideshares, blog posts from found.no, elasticseearch.org, etc - and
can't really pinpoint a way to improve our setup.

Thanks!

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/47b93b84-d929-4cad-becd-31581cd4c574%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/47b93b84-d929-4cad-becd-31581cd4c574%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_JtmLUJg7W3bc_7t%3Dti9Bb%2B7YFOnRzQQ6cbNAEKh2SMw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Hibernate Search with Elasticsearch

I ran this part in cmd and nothing happened after I did jetty:run in Spring
for branch00

# Create one person
curl -XPUT http://127.0.0.1:8080/api/1/person/1 -d '{name:David Pilato}'
# Read that person
curl http://127.0.0.1:8080/api/1/person/1

-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAHV%3D-PknGA%3DUDAfZau6GOcjhjZfMyKuZ6HtAsiSnf%3D_t8%2B7mTA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Tips for failing handshake with shield

Hi Jay, thanks again for the help. You were so right. Change the timeout to
30s and now the cluster is up and running. Now I can go for the next steps.

regards Jettro

On Fri, Feb 13, 2015 at 3:12 AM, Jay Modi jay.m...@elasticsearch.com
wrote:

 I think you are running into the TLS handshake taking too long and it
 causes the pings to timeout and then the Netty channels are closed, which
 is why you see the connection closed messages. Since you are running on a
 Raspberry PI, the overhead of TLS is going to be a lot for that platform;
 the handshake is expensive in terms of CPU. You probably need to increase
 the zen ping timeout and based on your other post it may need to be like
 over 25 seconds.


 On Thursday, February 12, 2015 at 11:20:35 AM UTC-8, Jettro Coenradie
 wrote:

 Thanks Jay for your answer. Had a good look at the appendix with
 troubleshooting, but cannot really find something to help me out. On the
 client I see this message:

 [2015-02-12 20:10:28,363][ERROR][shield.transport.netty   ] [jc-pi-glas]
 SSL/TLS handshake failed, closing channel: null

 There is a stack trace after this error:
 [2015-02-12 20:10:28,264][WARN ][shield.transport.netty   ] [jc-pi-glas]
 exception caught on transport layer [[id: 0x127799ce, /192.168.1.11:50409
 = 192.168.1.10/192.168.1.10:9300]
 http://192.168.1.10/192.168.1.10:9300%5D], closing connection
 java.lang.IllegalStateException: Internal error
 at sun.security.ssl.SSLEngineImpl.initHandshaker(SSLEngineImpl.java:464)
 at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1001)
 at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:901)
 at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:775)
 at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
 at org.elasticsearch.common.netty.handler.ssl.SslHandler.
 unwrap(SslHandler.java:1285)
 at org.elasticsearch.common.netty.handler.ssl.SslHandler.
 decode(SslHandler.java:917)
 at org.elasticsearch.common.netty.handler.codec.frame.
 FrameDecoder.callDecode(FrameDecoder.jav


 I also get numerous timeouts. The strange thing is that these timeout are
 pings to the node itself.

 [2015-02-12 20:10:25,151][WARN ][discovery.zen.ping.unicast] [jc-pi-glas]
 failed to send ping to [[jc-pi-glas][RBV7_u5yRsChF_
 M8Ep2ICQ][jc-pi-glas][inet[192.168.1.11/192.168.1.11:9300]
 http://192.168.1.11/192.168.1.11:9300%5D]]
 org.elasticsearch.transport.ReceiveTimeoutTransportException:
 [jc-pi-glas][inet[192.168.1.11/192.168.1.11:9300]][
 internal:discovery/zen/unicast
 http://192.168.1.11/192.168.1.11:9300%5D%5D%5Binternal:discovery/zen/unicast]
 request_id [5] timed out after [3751ms]
 at org.elasticsearch.transport.TransportService$TimeoutHandler.run(
 TransportService.java:366)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(
 ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(
 ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:744)


 On the red pi I get this message:
 [2015-02-12 20:10:44,245][WARN ][shield.transport.netty   ] [jc-pi-red]
 exception caught on transport layer [[id: 0xdc4d94e1, /192.168.1.11:50409
 = /192.168.1.10:9300]], closing connection
 javax.net.ssl.SSLException: Received close_notify during handshake
 at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
 at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1646)
 at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1614)

 I also created a very small java application with sockets and used the
 same jks files to authenticate. There it works.

 Any other ideas?


 On Wed, Feb 11, 2015 at 7:05 PM, Jay Modi jay@elasticsearch.com
 wrote:

 There should be another message on the other end of the connection that
 has more details of the actual failure. That message can be seen on the
 client side of the connection when the connection was closed by the server
 side.

 Also, please see http://www.elasticsearch.org/guide/en/shield/current/
 trouble-shooting.html#_sslhandshakeexception_causing_connections_to_fail
 for common problems and some tips on how to resolve them.

 On Wednesday, February 11, 2015 at 8:54:27 AM UTC-8, Jettro Coenradie
 wrote:

 Hi,
 I am trying to get SSL/TLS to work on my raspberry pi cluster. To bad I
 get the following message:

 [2015-02-11 17:51:35,037][ERROR][shield.transport.netty   ]
 [jc-pi-red] SSL/TLS handshake failed, closing channel: null

 So I guess something is wrong in my certificate chain. I have created
 my own CA and followed the steps from the getting started guide. Any tips
 on how to start debugging?

 thanks

 Jettro

  --
 You received this message because you are subscribed to a topic in the
 Google Groups elasticsearch group.
 To unsubscribe from this topic, visit https://groups.google.com/d/
 topic/elasticsearch/g-AT4CAVBCw/unsubscribe.
 To unsubscribe from this group and all its topics, send an email to
 elasticsearc...@googlegroups.com.
 To view this discussion on the web visit

Re: Hibernate Search with Elasticsearch

I ran jetty:run on branch00.

I think that happened because when I used curl -XPUT, nothing happened.
However, when I initialized a database and it produced results, I was able
to use the search function.

On Thu, Feb 12, 2015 at 9:57 PM, David Pilato da...@pilato.fr wrote:

CRUD=Create, Read, Update, Delete

No. What exact operation did you run?

David

Le 13 févr. 2015 à 01:22, William Yang williamyan...@gmail.com a écrit :

Ah I see, I will try to go through the other branches as well.

However, when I try to do the CRUD operations (What does that stand for by
the way?), I get a

16:20:55,055 DEBUG [PersonRestAPI] create(1, )

16:20:55,055 DEBUG [PersonService] get(1)=null

Do you know the cause of that?

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/e01521fd-fee5-4e21-b0c9-05cd50892fb5%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/e01521fd-fee5-4e21-b0c9-05cd50892fb5%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups elasticsearch group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/XfY9hJsKcOM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/4B3E165A-DC75-4462-BD46-A64739FFE710%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/4B3E165A-DC75-4462-BD46-A64739FFE710%40pilato.fr?utm_medium=emailutm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHV%3D-PkNO1FHDte0A2KwhYgOX0mZfPiVURRe-TOb7ncc_iDwyQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Hibernate Search with Elasticsearch

I meant what exact curl command you run?

David

Le 13 févr. 2015 à 07:17, William Yang williamyan...@gmail.com a écrit :

I ran jetty:run on branch00.

I think that happened because when I used curl -XPUT, nothing happened.
However, when I initialized a database and it produced results, I was able to
use the search function.

On Thu, Feb 12, 2015 at 9:57 PM, David Pilato da...@pilato.fr wrote:
CRUD=Create, Read, Update, Delete

No. What exact operation did you run?

David

Le 13 févr. 2015 à 01:22, William Yang williamyan...@gmail.com a écrit :

Ah I see, I will try to go through the other branches as well.

However, when I try to do the CRUD operations (What does that stand for by
the way?), I get a
16:20:55,055 DEBUG [PersonRestAPI] create(1, )

16:20:55,055 DEBUG [PersonService] get(1)=null

Do you know the cause of that?

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/e01521fd-fee5-4e21-b0c9-05cd50892fb5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHV%3D-PkNO1FHDte0A2KwhYgOX0mZfPiVURRe-TOb7ncc_iDwyQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/AFFA21AB-DE4E-4B9D-B2EA-053B71F2A439%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Re: Hibernate Search with Elasticsearch

Ok. But still the injection works? So you can start from here.
The REST part should not be a concern here as unrelated to your original 
question.

David

 Le 13 févr. 2015 à 08:07, William Yang williamyan...@gmail.com a écrit :
 
 I ran this part in cmd and nothing happened after I did jetty:run in Spring 
 for branch00
 
 # Create one person
 curl -XPUT http://127.0.0.1:8080/api/1/person/1 -d '{name:David Pilato}'
 
 # Read that person
 curl http://127.0.0.1:8080/api/1/person/1
 
 
 -- 
 You received this message because you are subscribed to the Google Groups 
 elasticsearch group.
 To unsubscribe from this group and stop receiving emails from it, send an 
 email to elasticsearch+unsubscr...@googlegroups.com.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/elasticsearch/CAHV%3D-PknGA%3DUDAfZau6GOcjhjZfMyKuZ6HtAsiSnf%3D_t8%2B7mTA%40mail.gmail.com.
 For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/D7157CC4-826B-493E-B5A4-13F3A5911206%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Re: Shield: Why do I get timeout on my own node

As pointed out by Jay Modi when discussing another question I had, this is
a timeout issue. The pi's do not have a lot of processing power and adding
ssl is to heavy for the pi. Upgrading the timeout to 30 seconds did help.

On Thu, Feb 12, 2015 at 9:41 PM, Jettro Coenradie
jettro.coenra...@gmail.com wrote:

Hi I am trying to create a cluster that is secured on a number of
raspberry pi's. If I start just one node with the following config I get a
timeout to that same node. So it is trying to ping itself and that does not
work. Any ideas?

Config:
cluster.name: jc-mypi
node.name: jc-pi-red
marvel.agent.indices: my*
marvel.agent.interval: 10s
marvel.agent.enabled: false
shield.audit.enabled: true
shield.ssl.keystore.path: /etc/elasticsearch/jcpi.jks
shield.ssl.keystore.password: geheim
shield.transport.ssl: true
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: [192.168.1.11:9300, 192.168.1.12:9300
]

In the log:

[2015-02-12 21:36:40,078][WARN ][shield.transport ] [jc-pi-red]
Received response for a request that has timed out, sent [21455ms] ago,
timed out [17704ms] ago, action [internal:discovery/zen/unicast], node
[[jc-pi-red][o4b8he9qQFiSzBV3ckkF1Q][jc-pi-red][inet[
192.168.1.10/192.168.1.10:9300]]
http://192.168.1.10/192.168.1.10:9300%5D%5D], id [0]

Thanks

--
You received this message because you are subscribed to a topic in the
Google Groups elasticsearch group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/9B3K7_J-hCg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/791792bc-9ee3-48ca-9f61-e2bd71952cc9%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/791792bc-9ee3-48ca-9f61-e2bd71952cc9%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Jettro Coenradie
http://www.gridshore.nl

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CABB4caPD3X9U9A%3DT_JirRA5nocanu-WfarJU6X%2BepHyLj0i5UA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: A strange behavior we've encountered on our ELK

Sort of... The ELK is running as a VM on a dedicated ESXi. Are there
special configurations I should do in such a case?

Thanks,
Yuval.

On Thursday, February 12, 2015, Itamar Syn-Hershko ita...@code972.com
wrote:

 Yes - can you try using the bulk API? Also, are you running on a cloud
 server?

 --

 Itamar Syn-Hershko
 http://code972.com | @synhershko https://twitter.com/synhershko
 Freelance Developer  Consultant
 Lucene.NET committer and PMC member

 On Thu, Feb 12, 2015 at 11:28 AM, Yuval Khalifa iyuv...@gmail.com
 javascript:_e(%7B%7D,'cvml','iyuv...@gmail.com'); wrote:

 Hi,

 I wrote that program and ran it and it did managed to keep a steady rate
 of about 1,000 events per minute even when the Kibana's total events per
 minute dropped from 60,000 to 6,000. However, when the
 Kibana's total events per minute dropped to zero, my program got a
 connection refused exception. I ran netstat -s and found out that every
 time the Kibana's line hit zero the number of RX-DRP increased. At that
 point I understood that I forgot to mention that this server has a 10GbE
 nic. Is it possible that the packets are being dropped because of some
 bufferis filling up? If so, how can I test it and verify that this is
 actually the case? If it is, how can I solve it?

 Thanks,
 Yuval.
 On Wednesday, February 11, 2015, Yuval Khalifa iyuv...@gmail.com
 javascript:_e(%7B%7D,'cvml','iyuv...@gmail.com'); wrote:

 Hi.

 When you say see how the file behaves I'm not quite sure what you
 mean by that... As I mentioned earlier, it's not that events do not appear
 at all but instead, the RATE at which they come decreases, so how can I
 measure the events rate in a file? I thought that there's another way that
 I can test this: I'll write a quick-and-dirty program that will send an
 event to the ELK via TCP every 12ms which should result in events rate of
 about 5,000 events per minute and I'll let you know if the events rate
 continues to drop or not...


 Thanks,
 Yuval.

 On Tuesday, February 10, 2015, Itamar Syn-Hershko ita...@code972.com
 wrote:

 I'd start by using logstash with input tcp and output fs and see how
 the file behaves. Same for the fs inputs - see how their files behave. And
 take it from there.

 --

 Itamar Syn-Hershko
 http://code972.com | @synhershko https://twitter.com/synhershko
 Freelance Developer  Consultant
 Lucene.NET committer and PMC member

 On Tue, Feb 10, 2015 at 7:47 PM, Yuval Khalifa iyuv...@gmail.com
 wrote:

 Great! How can I check that?


 On Tuesday, February 10, 2015, Itamar Syn-Hershko ita...@code972.com
 wrote:

 The graphic you sent suggests the issue is with logstash - since the
 @timestamp field is being populated by logstash and is the one that is 
 used
 to display the date histogram graphics in Kibana. I would start there. 
 I.e.
 maybe SecurityOnion buffers writes etc, and then to check the logstash
 shipper process stats.

 --

 Itamar Syn-Hershko
 http://code972.com | @synhershko https://twitter.com/synhershko
 Freelance Developer  Consultant
 Lucene.NET committer and PMC member

 On Tue, Feb 10, 2015 at 7:07 PM, Yuval Khalifa iyuv...@gmail.com
 wrote:

 Hi.

 Absolutely (but since that in the past I also worked at the helpdesk
 dept. I certainly understand why it is important to ask those Are you 
 sure
 it's plugged in? questions...). One of the logs is comming from
 SecurityOnion which logs (via bro-conn) all the connections so it must 
 be
 sending data 24x7x365.

 Thanks for the quick reply,
 Yuval.

 On Tuesday, February 10, 2015, Itamar Syn-Hershko 
 ita...@code972.com wrote:

 Are you sure your logs are generated linearly without bursts?

 --

 Itamar Syn-Hershko
 http://code972.com | @synhershko https://twitter.com/synhershko
 Freelance Developer  Consultant
 Lucene.NET committer and PMC member

 On Tue, Feb 10, 2015 at 6:29 PM, Yuval Khalifa iyuv...@gmail.com
 wrote:

 Hi,

 We just installed an ELK server and configured the logstash
 configuration to match the data that we send to it and until last 
 month it
 seems to be working fine but since then we see very strange behavior 
 in the
 Kibana, the event over time histogram shows the event rate at the 
 normal
 level for about a half an hour, then drops to about 20% of the normal 
 rate
 and then it continues to drop slowly for about two hours and then 
 stops and
 after a minute or two it returns to normal for the next half an hour 
 or so
 and the same behavior repeats. Needless to say that both the
 /var/log/logstash and /var/log/elasticsearch both show nothing since 
 the
 service started and by using tcpdump we can verify that events keep 
 coming
 in at the same rate all time. I attached our logstash configuration, 
 the
 /var/logstash/logstash.log, the 
 /var/log/elasticsearch/clustername.log and
 a screenshot of our Kibana with no filter applied so that you can see 
 the
 weird behavior that we see.

 Is there someone/somewhere that we can turn to to get some help on
 the subject?


 Thanks a lot,
 Yuval.

Master Node vs. Data Node Architecture

2015-02-12 Thread Eric

Hello,

Currently I have a 3 node ElasticSearch cluster. Each node is a RHEL VM
with 16 gig RAM. The basic config is:

- All nodes can be master and are data nodes.
- 3 shards and 1 replica
- 6 different indexes

My question is what can I do to help the stability and speed of my cluster.
Currently I'm having issues with 1 node going down and it taking everything
else down. The HA portion isn't working very well. I'm debating about
either adding 1 more node with the exact same stats or adding 2 more
smaller VMs that will act as master nodes only. I didn't know which one was
recommended or where I would get the biggest bang for the buck.

Any information would be greatly appreciated.

Thanks,
Eric

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/354a2326-5532-4239-87ea-f02af64fe71f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Master Node vs. Data Node Architecture

See this:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-node.html

Basically, the recommended pattern talks about isolating responsibilities.
A node should either be a data node, master-eligible node, or an external
gateway to the cluster (client node)

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 4:08 PM, Eric eric.luel...@gmail.com wrote:

Hello,

Currently I have a 3 node ElasticSearch cluster. Each node is a RHEL VM
with 16 gig RAM. The basic config is:

- All nodes can be master and are data nodes.
- 3 shards and 1 replica
- 6 different indexes

Any information would be greatly appreciated.

Thanks,
Eric

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/354a2326-5532-4239-87ea-f02af64fe71f%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/354a2326-5532-4239-87ea-f02af64fe71f%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZureLROJMaO7gVprFjA2OmRZA0ZYyH1v%2Bges06u_V__6w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

real one shot jdbc river

2015-02-12 Thread Abid Hussain

Hi all,

we run a jdbc river once per night for a complete recreation of our index.

The river - once it has been created - will run again after a restart of 
elasticsearch cluster (single node cluster). But what we try to achieve is 
a one shot river, which isn't being re-run after restart of cluster.

Naturally we can trigger an event (e.g. as a cronjob) which - after the 
estimated end of river run - deletes the river again. But this results in 
additional efforts.

The easiest strategy may be do delete the river right away after creating 
it. But I suppose in this case the running river would be stopped...?

*My question is: what is the best way to create a one shot river which is 
not being re-run after restart of cluster?*

I noticed the oneshot strategy but didn't find any documentation 
in https://github.com/jprante/elasticsearch-river-jdbc/wiki/Strategies.

Any help would be appreciated.

Regards,

Abid

-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d642e1ff-64a1-4ac9-b2b3-bba38b5c07e5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Moving to production considerations - applications with their clusters' infrastructure

Hi everyone,

I am considering moving one or several elasticsearch clusters to production.
Although Elasticsearch's documentation and community is *great*, I am 
strongly startled not to find any *complete use-case story* stretching from 
application(s) needs and data considerations to hardware ones.
Indeed, I understand why what/how much hardware / configuration / 
sharding questions are systematically replied with both it depends 
followed by test.
But then, what about a few complete descriptions, out of so many 
elasticsearch users, from data use case to cluster's internals, along with 
a few performance and nodes stats?

So here are questions, before moving to production :

Are there any *complete* use cases around? Could you share some? By 
complete I mean including *at least some* of the following :
   
   1. *Application needs and scope*
   2. *Indexing Data indications* : data volume, documents mapping, 
   documents / indexes volume
   3. *Searching Data indications* : different applications, queries, use 
   of facets - filters - aggregations, concurrent indexing
   4. *Cluster Hardware* : machines' hardware (RAM, Disks/SSD - 
   DAS-JBOD/SAN/NAS), JVM heap / OS Cache, nb of machines, back office network
   5. *Cluster Configuration* : one or several indexes, sharding, 
   replication, master nodes, data nodes, use of over-sharding at start-up, 
   use of re-indexing
   6. *Benchmaks *: queries response times, QPS, with or without concurrent 
   indexing, memory heap sweet spot, nodes stats

For those interested, here are the (not *complete*) best-among-very-few 
exemples I've stumbled upon so far :

   - The very best (perfs with hardware and query description) : 
   
http://fr.slideshare.net/charliejuggler/lucene-solrlondonug-meetup28nov2014-solr-es-performance
   - Hardware and master nodes heap : 
   
https://groups.google.com/forum/?fromgroups#!searchin/elasticsearch/sizing/elasticsearch/V5BtrCGOqoU/l7x6vqMEx5YJ
   - *6th slide* - Hardware and storage with number of documents (well, 
   without indexes and documents storage volume nor RAM consumption) :
   
   https://speakerdeck.com/bhaskarvk/scaling-elasticsearch-washington-dc-meetup
   With JBOD / SAN storage discussion in To Raid or not to Raid:
   
   
https://groups.google.com/forum/?fromgroups#!searchin/elasticsearch/hardware/elasticsearch/HSj2fZGdU1Y/4mFCBTCb-JcJ
   - Usual heap considerations in a real case :
   
   
https://codeascraft.com/2014/12/04/juggling-multiple-elasticsearch-instances-on-a-single-host/

Do not forget Elasticsearch awesome docs for moving to production 
considerations :

   - 
   
http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/administration.html
   - 
   http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/deploy.html
   - 
   
*http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/hardware.html*
   - 
   
*http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/heap-sizing.html*


-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/927f60b1-8ae2-463e-b725-5f4f993905d9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Use cases - Production examples: datas, queries, cluster hardware and conf, and statistics

Hi everyone,

I am considering moving one or several elasticsearch clusters to production.
Although Elasticsearch's documentation and community is *great*, I am 
strongly startled not to find any *complete use-case story* stretching from 
application(s) needs and data considerations to hardware ones.
Indeed, I understand why what/how much hardware / configuration / 
sharding questions are systematically replied with both it depends 
followed by test.
But then, what about a few complete descriptions, out of so many 
elasticsearch users, from data use case to cluster's internals, along with 
a few performance and nodes stats?

So here are questions, before moving to production :

Are there any *complete* use cases around? Could you share some? By 
complete I mean including *at least some* of the following :
   
   1. *Application needs and scope*
   2. *Indexing Data indications* : data volume, documents mapping, 
   documents / indexes volume
   3. *Searching Data indications* : different applications, queries, use 
   of facets - filters - aggregations, concurrent indexing
   4. *Cluster Hardware* : machines' hardware (RAM, Disks/SSD - 
   DAS-JBOD/SAN/NAS), JVM heap / OS Cache, nb of machines, back office network
   5. *Cluster Configuration* : one or several indexes, sharding, 
   replication, master nodes, data nodes, use of over-sharding at start-up, 
   use of re-indexing
   6. *Benchmaks *: queries response times, QPS, with or without concurrent 
   indexing, memory heap sweet spot, nodes stats

For those interested, here are the (not *complete*) best-among-very-few 
exemples I've stumbled upon so far :

   - The very best (perfs with hardware and query description) : 
   
http://fr.slideshare.net/charliejuggler/lucene-solrlondonug-meetup28nov2014-solr-es-performance
   - Hardware and master nodes heap : 
   
https://groups.google.com/forum/?fromgroups#!searchin/elasticsearch/sizing/elasticsearch/V5BtrCGOqoU/l7x6vqMEx5YJ
   - *6th slide* - Hardware and storage with number of documents (well, 
   without indexes and documents storage volume nor RAM consumption) :
   
   https://speakerdeck.com/bhaskarvk/scaling-elasticsearch-washington-dc-meetup
   With JBOD / SAN storage discussion in To Raid or not to Raid:
   
   
https://groups.google.com/forum/?fromgroups#!searchin/elasticsearch/hardware/elasticsearch/HSj2fZGdU1Y/4mFCBTCb-JcJ
   - Usual heap considerations in a real case :
   
   
https://codeascraft.com/2014/12/04/juggling-multiple-elasticsearch-instances-on-a-single-host/

Do not forget Elasticsearch awesome docs for moving to production 
considerations :

   - 
   
http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/administration.html
   - 
   http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/deploy.html
   - 
*http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/hardware.html
 
   
http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/hardware.html*
   - 
*http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/heap-sizing.html
 
   
http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/heap-sizing.html*

-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/57c967ea-8bf0-4dce-a7ca-4a746ee21250%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Multy-tenancy - level of service garantee

Hi everyone,

After my precedent question 
https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/US-BA4R_Qdc 
regarding examples of clusters in production, I am wondering about 
multy-tenancy and garantee of service in Elasticsearch :

*Multy-tenant cluster* : Is there a way to *garantee a level of service* / 
capacity planning for *each tenant* using the cluster (its *own indexes*) ?


Thanks,
 

-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/7f99a562-ae18-447f-b227-cd145483dcf3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch + attachment plugin + Kibana + couchbase

The XDCR plugin indexes the data using an envelope document. Long story
short, make sure you use the latest XDCR plugin as older ones are missing
lots of important functions, and use templates and dynamic templates with
proper field paths for this to work correctly

http://code972.com/blog/2015/02/80-elasticsearch-one-tip-a-day-managing-index-mappings-like-a-pro
http://code972.com/blog/2015/02/81-elasticsearch-one-tip-a-day-using-dynamic-templates-to-avoid-rigorous-mappings

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 3:59 PM, Nadav Hashimshony nad...@gmail.com wrote:

Hi,

I'm new to the group, hope ill find what i need and share my experience as
i go along..

im using ES with the attachment-plugin in order to store and search files.
when i set the mapping right and insert the file data in a Base64 manner
I'm able to query my data via Kibana.

my problem is this.

if i create the index + mapping in ES, then insert the data to Couchbase
and use XDRC to replicate it to ES, i can't query the Data with Kibana.
it looks like the mapping of the index created in ES doesn't index well
the data it gets from Couchbase.

has anyone encounter such an issue?

Thanks You

Nadav.

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8092eaf5-0ef8-4249-8e5d-acff8281a81a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8092eaf5-0ef8-4249-8e5d-acff8281a81a%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zs5OwXJe9aT1pPNu9vuooXO10Z3Mx7xc8CJh77EN9s%3DCQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Different ES clusters in the same host..

2015-02-12 Thread Ricardo Reis

Hello David,

It's really solve my problem ;)

Thanks

Ricardo Reis

Em sexta-feira, 6 de fevereiro de 2015 08:52:03 UTC-2, David Pilato 
escreveu:

 What we do this in AWS plugin:

 - get the IP address from AWS API (which obviously does know anything 
 about the port used)
 address = instance.getPrivateIpAddress();

 - transform this String to a TransportAddress which will basically be 
 IP:9300:
 TransportAddress[] addresses 
 = transportService.addressesFromString(address);

 - try to join IP:9300 which obviously answers but this is not the one you 
 are expecting.

 I wonder if setting transport.tcp.port: 9302 could help here.
 By default this value is 9300-9399.


 -- 
 *David Pilato* | *Technical Advocate* | *Elasticsearch.com 
 http://Elasticsearch.com*
 @dadoonet https://twitter.com/dadoonet | @elasticsearchfr 
 https://twitter.com/elasticsearchfr | @scrutmydocs 
 https://twitter.com/scrutmydocs


  
 Le 6 févr. 2015 à 01:23, Ricardo Alves dos Reis ricard...@gmail.com 
 javascript: a écrit :

 Hello folks,


 First of all, I'd like to explain my current necessity to bring this 
 future trend out, some projects in my company have bean developed to use a 
 lot of micro-services, unfortunately, is not uncommon that the mean of cpu 
 usage in our cloud servers (AWS) are sustained in a lower degrees.


 To provisioning these resources easily all the components are executed in 
 a docker containers and these stacks were represented as yaml file (
 http://www.fig.sh). In a group of servers I've more than one 
 Elasticsearch clusters, containerized to use EC2 discovery and with a 
 transport.publish_port configured to avoid collision, however, only the 
 first cluster in default transport port have a correct cluster formation.


 My guess is the transport.publish_post is not used as pointer to 
 discovery.zen.unicast to avoid unused transport port.. 


 Thanks for any kind of help.


 #docker #aws #discovery


 fig.yml


 esemail:

 image: XXX/elasticsearch:latest

 volumes:

 - /dev/log:/dev/log

 - /data0/email:/data/0

 - /data1/email:/data/1

 ports:

 - 9202:9200

 - 9302:9300

 environment:

 TRANSPORT_PUBLISH_PORT: 9302

 NETWORK_PUBLISH_HOST: _ec2_

 CLUSTER_ROUTING_ALLOCATION_SAME_SHARD_HOST: true

 ES_ROOT_LOG_LEVEL: INFO

 ES_DISCOVERY_LOG_LEVEL: TRACE

 ES_LOG_DIR: /var/log/elasticsearch

 ES_HEAP_SIZE: 1g

 ES_DIRECT_SIZE: 256m

 ES_JAVA_OPTS: -server -XX:+UseCompressedOops -XX:+DoEscapeAnalysis 
 -XX:+AlwaysPreTouch -verbose:gc -Xloggc:/var/log/elasticsearch/gc.log 
 -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=512K

 MAX_OPEN_FILES: 65535

 MAX_MAP_COUNT: 262144

 CLUSTER_NAME: email-es

 NODE_MASTER: true

 NODE_DATA: true

 PATH_DATA: /data/0/datasearch0,/data/1/datasearch0

 BOOTSTRAP_MLOCKALL: false

 DISCOVERY_ZEN_MCAST: false

 DISCOVERY_ZEN_REJOIN_ON_MASTER_GONE: false

 DISCOVERY_ZEN_MINIMUM_MASTER_NODES: 2

 DISCOVERY_TYPE: ec2

 DISCOVERY_EC2_ANY_GROUP: false

 DISCOVERY_EC2_GROUPS: db-prod-es-mail

 DISCOVERY_EC2_AVAILABILITY_ZONES: us-east-1a,us-east-1b,us-east-1d

 DISCOVERY_EC2_HOST_TYPE: private_ip

 DISCOVERY_EC2_PING_TIMEOUT: 1m

 GATEWAY_RECOVER_AFTER_NODES: 2

 GATEWAY_RECOVER_AFTER_TIME: 2m

 GATEWAY_EXPECTED_NODES: 3

 AWS_ELB_NAME: ESEmailBionexo

 AWS_ACCESS_KEY_ID: X

 AWS_SECRET_ACCESS_KEY:  

 AWS_DEFAULT_REGION: us-east-1

 THREAD_POOL_SEARCH: 2000

 THREAD_POOL_INDEX: 2000

 THREAD_POOL_BULK: 2000


 [ec2-user@ip-172-16-255-20 ~]$ sudo docker ps |grep elastic

 d1b9ebf8c605 bionexo/elasticsearch:2015020502 /bin/sh -c /bionexo 3 hours 
 ago Up 3 hours 0.0.0.0:9202-9200/tcp, 0.0.0.0:9302-9300/tcp 
 elasticsearch_esemail_1 

 088da31f943b bionexo/elasticsearch:2015020401 /bin/sh -c /bionexo 35 
 hours ago Up 35 hours 0.0.0.0:9200-9200/tcp, 0.0.0.0:9300-9300/tcp 
 elasticsearch_eslogger_1 


 [2015-02-05 20:53:09,834][DEBUG][action.admin.indices.create] 
 [d1b9ebf8c605] no known master node, scheduling a retry

 [2015-02-05 20:53:09,853][DEBUG][action.admin.indices.create] 
 [d1b9ebf8c605] observer: timeout notification from cluster service. timeout 
 setting [1m], time since start [1m]

 [2015-02-05 20:53:09,854][DEBUG][action.admin.indices.create] 
 [d1b9ebf8c605] observer: timeout notification from cluster service. timeout 
 setting [1m], time since start [1m]

 [2015-02-05 20:53:10,637][DEBUG][action.admin.indices.create] 
 [d1b9ebf8c605] no known master node, scheduling a retry

 [2015-02-05 20:53:10,637][DEBUG][action.admin.indices.create] 
 [d1b9ebf8c605] no known master node, scheduling a retry

 [2015-02-05 20:53:23,833][DEBUG][action.admin.indices.create] 
 [d1b9ebf8c605] no known master node, scheduling a retry

 [2015-02-05 20:53:25,837][DEBUG][action.admin.indices.create] 
 [d1b9ebf8c605] observer: timeout notification from cluster service. timeout 
 setting [1m], time since start [1m]

 [2015-02-05 20:53:26,618][DEBUG][action.admin.indices.create] 
 [d1b9ebf8c605] no known master node, scheduling a retry

 [2015-02-05

Re: Delete index after backup

2015-02-12 Thread Yarden Bar

Found the last section here
http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/retiring-data.html

'archiving old indices' to answer my question.

On Wednesday, February 11, 2015 at 5:17:04 PM UTC+2, Yarden Bar wrote:

but will the index deletion get reflected in the next snapshot invocation?

On Wednesday, February 11, 2015 at 5:11:23 PM UTC+2, David Pilato wrote:

Removing an index does not remove a snapshot.

--
*David Pilato* | *Technical Advocate* | *Elasticsearch.com
http://Elasticsearch.com*
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr
https://twitter.com/elasticsearchfr | @scrutmydocs
https://twitter.com/scrutmydocs

Le 11 févr. 2015 à 16:07, Yarden Bar ayash@gmail.com a écrit :

HI all,

Is there any procedure to *archive* index to HDFS (or other repo) and
delete it?

I've looked into the 'Snapshot/Restore' docs and understood that if I
execute snapshot - delete index - snapshot the index deletion will get
reflected in the snapshot process.

Thanks,
Yarden

https://groups.google.com/d/msgid/elasticsearch/2fca7f58-4fc3-475d-92dd-3d06b486b81d%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f9fb9951-a147-42a5-9048-8611d4f2b60e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Updating several indices at once

2015-02-12 Thread Aldian

Hi

Today I tried the following request, which worked great:
curl -XPUT 'localhost:9200/logstash-2014.03.24/_settings' -d '
{
index : {
number_of_replicas : 0
}
}'

Now I want to set number of replicas for every indice in 2014 to 0. Can I 
use a wildcard in the url such as the following? :
curl -XPUT 'localhost:9200/logstash-2014.*/_settings' -d '
{
index : {
number_of_replicas : 0
}
}'

Or do I have to make a shell script that will iterate over all the dates?

Thanks for your help

Aldian

-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/174155d7-6133-40f0-9d8f-c340971f1f44%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: A strange behavior we've encountered on our ELK

Yes, make sure the disk is local and not low latency shared one (e.g. SAN).
Also SSD will probably fix all your pains.

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer  Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 3:28 PM, Yuval Khalifa iyuv...@gmail.com wrote:

 Sort of... The ELK is running as a VM on a dedicated ESXi. Are there
 special configurations I should do in such a case?

 Thanks,
 Yuval.

 On Thursday, February 12, 2015, Itamar Syn-Hershko ita...@code972.com
 wrote:

 Yes - can you try using the bulk API? Also, are you running on a cloud
 server?

 --

 Itamar Syn-Hershko
 http://code972.com | @synhershko https://twitter.com/synhershko
 Freelance Developer  Consultant
 Lucene.NET committer and PMC member

 On Thu, Feb 12, 2015 at 11:28 AM, Yuval Khalifa iyuv...@gmail.com
 wrote:

 Hi,

 I wrote that program and ran it and it did managed to keep a steady rate
 of about 1,000 events per minute even when the Kibana's total events per
 minute dropped from 60,000 to 6,000. However, when the
 Kibana's total events per minute dropped to zero, my program got a
 connection refused exception. I ran netstat -s and found out that every
 time the Kibana's line hit zero the number of RX-DRP increased. At that
 point I understood that I forgot to mention that this server has a 10GbE
 nic. Is it possible that the packets are being dropped because of some
 bufferis filling up? If so, how can I test it and verify that this is
 actually the case? If it is, how can I solve it?

 Thanks,
 Yuval.
 On Wednesday, February 11, 2015, Yuval Khalifa iyuv...@gmail.com
 wrote:

 Hi.

 When you say see how the file behaves I'm not quite sure what you
 mean by that... As I mentioned earlier, it's not that events do not appear
 at all but instead, the RATE at which they come decreases, so how can I
 measure the events rate in a file? I thought that there's another way that
 I can test this: I'll write a quick-and-dirty program that will send an
 event to the ELK via TCP every 12ms which should result in events rate of
 about 5,000 events per minute and I'll let you know if the events rate
 continues to drop or not...


 Thanks,
 Yuval.

 On Tuesday, February 10, 2015, Itamar Syn-Hershko ita...@code972.com
 wrote:

 I'd start by using logstash with input tcp and output fs and see how
 the file behaves. Same for the fs inputs - see how their files behave. And
 take it from there.

 --

 Itamar Syn-Hershko
 http://code972.com | @synhershko https://twitter.com/synhershko
 Freelance Developer  Consultant
 Lucene.NET committer and PMC member

 On Tue, Feb 10, 2015 at 7:47 PM, Yuval Khalifa iyuv...@gmail.com
 wrote:

 Great! How can I check that?


 On Tuesday, February 10, 2015, Itamar Syn-Hershko ita...@code972.com
 wrote:

 The graphic you sent suggests the issue is with logstash - since the
 @timestamp field is being populated by logstash and is the one that is 
 used
 to display the date histogram graphics in Kibana. I would start there. 
 I.e.
 maybe SecurityOnion buffers writes etc, and then to check the logstash
 shipper process stats.

 --

 Itamar Syn-Hershko
 http://code972.com | @synhershko https://twitter.com/synhershko
 Freelance Developer  Consultant
 Lucene.NET committer and PMC member

 On Tue, Feb 10, 2015 at 7:07 PM, Yuval Khalifa iyuv...@gmail.com
 wrote:

 Hi.

 Absolutely (but since that in the past I also worked at
 the helpdesk dept. I certainly understand why it is important to ask 
 those
 Are you sure it's plugged in? questions...). One of the logs is 
 comming
 from SecurityOnion which logs (via bro-conn) all the connections so it 
 must
 be sending data 24x7x365.

 Thanks for the quick reply,
 Yuval.

 On Tuesday, February 10, 2015, Itamar Syn-Hershko 
 ita...@code972.com wrote:

 Are you sure your logs are generated linearly without bursts?

 --

 Itamar Syn-Hershko
 http://code972.com | @synhershko https://twitter.com/synhershko
 Freelance Developer  Consultant
 Lucene.NET committer and PMC member

 On Tue, Feb 10, 2015 at 6:29 PM, Yuval Khalifa iyuv...@gmail.com
 wrote:

 Hi,

 We just installed an ELK server and configured the logstash
 configuration to match the data that we send to it and until last 
 month it
 seems to be working fine but since then we see very strange behavior 
 in the
 Kibana, the event over time histogram shows the event rate at the 
 normal
 level for about a half an hour, then drops to about 20% of the 
 normal rate
 and then it continues to drop slowly for about two hours and then 
 stops and
 after a minute or two it returns to normal for the next half an hour 
 or so
 and the same behavior repeats. Needless to say that both the
 /var/log/logstash and /var/log/elasticsearch both show nothing since 
 the
 service started and by using tcpdump we can verify that events keep 
 coming
 in at the same rate all time. I attached our logstash configuration, 
 the
 /var/logstash/logstash.log, the

Elasticsearch + attachment plugin + Kibana + couchbase

Hi, 

I'm new to the group, hope ill find what i need and share my experience as 
i go along..

im using ES with the attachment-plugin in order to store and search files.
when i set the mapping right and insert the file data in a Base64 manner 
I'm able to query my data via Kibana.

my problem is this.

if i create the index + mapping in ES, then insert the data to Couchbase 
and use XDRC to replicate it to ES, i can't query the Data with Kibana.
it looks like the mapping of the index created in ES doesn't index well the 
data it gets from Couchbase.

has anyone encounter such an issue?

Thanks You 

Nadav.

-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/8092eaf5-0ef8-4249-8e5d-acff8281a81a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: A strange behavior we've encountered on our ELK

There's a good writeup on the subject by Mike btw, you should read it
http://www.elasticsearch.org/blog/performance-considerations-elasticsearch-indexing/

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer  Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 3:30 PM, Itamar Syn-Hershko ita...@code972.com
wrote:

 Yes, make sure the disk is local and not low latency shared one (e.g.
 SAN). Also SSD will probably fix all your pains.

 --

 Itamar Syn-Hershko
 http://code972.com | @synhershko https://twitter.com/synhershko
 Freelance Developer  Consultant
 Lucene.NET committer and PMC member

 On Thu, Feb 12, 2015 at 3:28 PM, Yuval Khalifa iyuv...@gmail.com wrote:

 Sort of... The ELK is running as a VM on a dedicated ESXi. Are there
 special configurations I should do in such a case?

 Thanks,
 Yuval.

 On Thursday, February 12, 2015, Itamar Syn-Hershko ita...@code972.com
 wrote:

 Yes - can you try using the bulk API? Also, are you running on a cloud
 server?

 --

 Itamar Syn-Hershko
 http://code972.com | @synhershko https://twitter.com/synhershko
 Freelance Developer  Consultant
 Lucene.NET committer and PMC member

 On Thu, Feb 12, 2015 at 11:28 AM, Yuval Khalifa iyuv...@gmail.com
 wrote:

 Hi,

 I wrote that program and ran it and it did managed to keep a steady
 rate of about 1,000 events per minute even when the Kibana's total events
 per minute dropped from 60,000 to 6,000. However, when the
 Kibana's total events per minute dropped to zero, my program got a
 connection refused exception. I ran netstat -s and found out that every
 time the Kibana's line hit zero the number of RX-DRP increased. At that
 point I understood that I forgot to mention that this server has a 10GbE
 nic. Is it possible that the packets are being dropped because of some
 bufferis filling up? If so, how can I test it and verify that this is
 actually the case? If it is, how can I solve it?

 Thanks,
 Yuval.
 On Wednesday, February 11, 2015, Yuval Khalifa iyuv...@gmail.com
 wrote:

 Hi.

 When you say see how the file behaves I'm not quite sure what you
 mean by that... As I mentioned earlier, it's not that events do not appear
 at all but instead, the RATE at which they come decreases, so how can I
 measure the events rate in a file? I thought that there's another way that
 I can test this: I'll write a quick-and-dirty program that will send an
 event to the ELK via TCP every 12ms which should result in events rate of
 about 5,000 events per minute and I'll let you know if the events rate
 continues to drop or not...


 Thanks,
 Yuval.

 On Tuesday, February 10, 2015, Itamar Syn-Hershko ita...@code972.com
 wrote:

 I'd start by using logstash with input tcp and output fs and see how
 the file behaves. Same for the fs inputs - see how their files behave. 
 And
 take it from there.

 --

 Itamar Syn-Hershko
 http://code972.com | @synhershko https://twitter.com/synhershko
 Freelance Developer  Consultant
 Lucene.NET committer and PMC member

 On Tue, Feb 10, 2015 at 7:47 PM, Yuval Khalifa iyuv...@gmail.com
 wrote:

 Great! How can I check that?


 On Tuesday, February 10, 2015, Itamar Syn-Hershko 
 ita...@code972.com wrote:

 The graphic you sent suggests the issue is with logstash - since
 the @timestamp field is being populated by logstash and is the one 
 that is
 used to display the date histogram graphics in Kibana. I would start 
 there.
 I.e. maybe SecurityOnion buffers writes etc, and then to check the 
 logstash
 shipper process stats.

 --

 Itamar Syn-Hershko
 http://code972.com | @synhershko https://twitter.com/synhershko
 Freelance Developer  Consultant
 Lucene.NET committer and PMC member

 On Tue, Feb 10, 2015 at 7:07 PM, Yuval Khalifa iyuv...@gmail.com
 wrote:

 Hi.

 Absolutely (but since that in the past I also worked at
 the helpdesk dept. I certainly understand why it is important to ask 
 those
 Are you sure it's plugged in? questions...). One of the logs is 
 comming
 from SecurityOnion which logs (via bro-conn) all the connections so 
 it must
 be sending data 24x7x365.

 Thanks for the quick reply,
 Yuval.

 On Tuesday, February 10, 2015, Itamar Syn-Hershko 
 ita...@code972.com wrote:

 Are you sure your logs are generated linearly without bursts?

 --

 Itamar Syn-Hershko
 http://code972.com | @synhershko https://twitter.com/synhershko
 Freelance Developer  Consultant
 Lucene.NET committer and PMC member

 On Tue, Feb 10, 2015 at 6:29 PM, Yuval Khalifa iyuv...@gmail.com
  wrote:

 Hi,

 We just installed an ELK server and configured the logstash
 configuration to match the data that we send to it and until last 
 month it
 seems to be working fine but since then we see very strange 
 behavior in the
 Kibana, the event over time histogram shows the event rate at the 
 normal
 level for about a half an hour, then drops to about 20% of the 
 normal rate
 and then it continues to drop slowly for about two hours and then 
 stops and

Re: performance getting even worse after optimization

2015-02-12 Thread MohammadReza Zolfaghari

Hi Xiaoting Ye,

Do you find a solution for your problem?

Best,

On Thursday, January 8, 2015 at 9:39:16 PM UTC+3:30, Xiaoting Ye wrote:

The index is of 149G, 19 shards with 1 replica.

The es version is 1.4.1, and the java version is 1.7.0_71.

I have specific routing strategy and the query used in testing only goes
to one shard:
heap.percentram.percent load
5158 0.33

(when it is under continues query, just one query at a time)

This specific shard has 22502484 docs, 10GB in size.

Thanks!

On Thu, Jan 8, 2015 at 2:10 AM, Mark Walkom markw...@gmail.com
javascript: wrote:

How big is the index, how many shards and replicas?
What ES version? What java version?

On 8 January 2015 at 20:40, Xiaoting Ye x...@spokeo.com javascript:
wrote:

Hi,

I just did an _optimize operation on a cluster (10 data nodes, roughly
350,000,000 docs in total). This a cluster only has one index.

However, the performance gets even worse: the response time doubled or
even tripled.

Any hint on this?

Thanks!

https://groups.google.com/d/msgid/elasticsearch/d9f9ba25-4a7f-4fba-978c-8368d74bc349%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups elasticsearch group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/W49B4d9MWNk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearc...@googlegroups.com javascript:.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8B_qPdBA0S8JeXmhM2e013-YxQb5roZAJvEh-r1rxfQQ%40mail.gmail.com

https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8B_qPdBA0S8JeXmhM2e013-YxQb5roZAJvEh-r1rxfQQ%40mail.gmail.com?utm_medium=emailutm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/542cdeb0-2f76-4995-9343-cd64ac62b1a9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: real one shot jdbc river

2015-02-12 Thread Abid Hussain

So I tried the oneshot strategy with no success. The river is being rerun
after restart of cluster.

Am Donnerstag, 12. Februar 2015 15:22:27 UTC+1 schrieb Abid Hussain:

Hi all,

we run a jdbc river once per night for a complete recreation of our index.

The river - once it has been created - will run again after a restart of
elasticsearch cluster (single node cluster). But what we try to achieve is
a one shot river, which isn't being re-run after restart of cluster.

Naturally we can trigger an event (e.g. as a cronjob) which - after the
estimated end of river run - deletes the river again. But this results in
additional efforts.

The easiest strategy may be do delete the river right away after creating
it. But I suppose in this case the running river would be stopped...?

*My question is: what is the best way to create a one shot river which is
not being re-run after restart of cluster?*

I noticed the oneshot strategy but didn't find any documentation in
https://github.com/jprante/elasticsearch-river-jdbc/wiki/Strategies.

Any help would be appreciated.

Regards,

Abid

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/dea5b9f2-36a5-4f09-b2c0-ca44157a4d9c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: real one shot jdbc river

2015-02-12 Thread joergpra...@gmail.com

Yes, a river instance is restarted by ES automatically at node restart. You
must delete it after work is done in order to free resources.

Just a note, in future versions, I will drop the ES river API, and the JDBC
plugin will manage an internal state, where the number of execution cycles
can be configured.

Jörg

On Thu, Feb 12, 2015 at 5:18 PM, Abid Hussain huss...@novacom.mygbiz.com
wrote:

So I tried the oneshot strategy with no success. The river is being rerun
after restart of cluster.

Am Donnerstag, 12. Februar 2015 15:22:27 UTC+1 schrieb Abid Hussain:

Hi all,

we run a jdbc river once per night for a complete recreation of our index.

Naturally we can trigger an event (e.g. as a cronjob) which - after the
estimated end of river run - deletes the river again. But this results in
additional efforts.

The easiest strategy may be do delete the river right away after creating
it. But I suppose in this case the running river would be stopped...?

*My question is: what is the best way to create a one shot river which is
not being re-run after restart of cluster?*

I noticed the oneshot strategy but didn't find any documentation in
https://github.com/jprante/elasticsearch-river-jdbc/wiki/Strategies.

Any help would be appreciated.

Regards,

Abid

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/dea5b9f2-36a5-4f09-b2c0-ca44157a4d9c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/dea5b9f2-36a5-4f09-b2c0-ca44157a4d9c%40googlegroups.com?utm_medium=emailutm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHD2VbNz78YwzLK6iETXjYGCEtXyiwYmhBrXr2AsGvr0A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch + attachment plugin + Kibana + couchbase

Like I said, you need the mapping to catch before the XDCR plugin begins
the replication - so you need to put a template with this mapping that will
override XDCR's

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 4:59 PM, Nadav Hashimshony nad...@gmail.com wrote:

Thanks you for the response

i am using mapping, i created the following index
PUT /storage/files/_mapping
{
files: {
properties: {
file: {
type: attachment,
path: full,
fields: {
content_type: {
type: string,
store: true
}
}
}
}
}
}

when i insert data via ES and query it, all is fine.
the problem is when data is inserted to Couchbase..

Nadav

On Thursday, February 12, 2015 at 4:03:01 PM UTC+2, Itamar Syn-Hershko
wrote:

http://code972.com/blog/2015/02/80-elasticsearch-one-tip-a-
day-managing-index-mappings-like-a-pro
http://code972.com/blog/2015/02/81-elasticsearch-one-tip-a-
day-using-dynamic-templates-to-avoid-rigorous-mappings

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 3:59 PM, Nadav Hashimshony nad...@gmail.com
wrote:

Hi,

I'm new to the group, hope ill find what i need and share my experience
as i go along..

im using ES with the attachment-plugin in order to store and search
files.
when i set the mapping right and insert the file data in a Base64 manner
I'm able to query my data via Kibana.

my problem is this.

has anyone encounter such an issue?

Thanks You

Nadav.

--
You received this message because you are subscribed to the Google
Groups elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/8092eaf5-0ef8-4249-8e5d-acff8281a81a%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8092eaf5-0ef8-4249-8e5d-acff8281a81a%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1d9c8ce5-116f-40cc-a5e3-6ebe47191850%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/1d9c8ce5-116f-40cc-a5e3-6ebe47191850%40googlegroups.com?utm_medium=emailutm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zuh7HHK8XmdznuHnw7E01ffXV8BC-49D70ekMc1-YhQCA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch + attachment plugin + Kibana + couchbase

ok, just to be clear.

the steps i did was as followed:
1. create the index with the mapping.
2. define the XDCR to replicate my bucket with the index in ES.
3. insert data to couchbase.
4. try to query with kibana

What you suggest is to Add another BEFORE step 1:
0. create a template to include my mapping.
1. crate the index in ES
and so on...

did i get it right?

Thanks.
Nadav.

On Thursday, February 12, 2015 at 5:04:24 PM UTC+2, Itamar Syn-Hershko
wrote:

Like I said, you need the mapping to catch before the XDCR plugin begins
the replication - so you need to put a template with this mapping that will
override XDCR's

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 4:59 PM, Nadav Hashimshony nad...@gmail.com
javascript: wrote:

Thanks you for the response

when i insert data via ES and query it, all is fine.
the problem is when data is inserted to Couchbase..

Nadav

On Thursday, February 12, 2015 at 4:03:01 PM UTC+2, Itamar Syn-Hershko
wrote:

http://code972.com/blog/2015/02/80-elasticsearch-one-tip-a-
day-managing-index-mappings-like-a-pro
http://code972.com/blog/2015/02/81-elasticsearch-one-tip-a-
day-using-dynamic-templates-to-avoid-rigorous-mappings

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 3:59 PM, Nadav Hashimshony nad...@gmail.com
wrote:

Hi,

I'm new to the group, hope ill find what i need and share my experience
as i go along..

im using ES with the attachment-plugin in order to store and search
files.
when i set the mapping right and insert the file data in a Base64
manner I'm able to query my data via Kibana.

my problem is this.

if i create the index + mapping in ES, then insert the data to
Couchbase and use XDRC to replicate it to ES, i can't query the Data with
Kibana.
it looks like the mapping of the index created in ES doesn't index well
the data it gets from Couchbase.

has anyone encounter such an issue?

Thanks You

Nadav.

--
You received this message because you are subscribed to the Google
Groups elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/8092eaf5-0ef8-4249-8e5d-acff8281a81a%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8092eaf5-0ef8-4249-8e5d-acff8281a81a%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

https://groups.google.com/d/msgid/elasticsearch/1d9c8ce5-116f-40cc-a5e3-6ebe47191850%40googlegroups.com?utm_medium=emailutm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8628ef07-2e10-423a-9de0-13ebaa37a0e8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch + attachment plugin + Kibana + couchbase

Yes. Just make sure the template reflects the actual document structure -
as I said XDCR wraps your document in an envelope document

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 5:12 PM, Nadav Hashimshony nad...@gmail.com wrote:

ok, just to be clear.

the steps i did was as followed:
1. create the index with the mapping.
2. define the XDCR to replicate my bucket with the index in ES.
3. insert data to couchbase.
4. try to query with kibana

What you suggest is to Add another BEFORE step 1:
0. create a template to include my mapping.
1. crate the index in ES
and so on...

did i get it right?

Thanks.
Nadav.

On Thursday, February 12, 2015 at 5:04:24 PM UTC+2, Itamar Syn-Hershko
wrote:

Like I said, you need the mapping to catch before the XDCR plugin
begins the replication - so you need to put a template with this mapping
that will override XDCR's

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 4:59 PM, Nadav Hashimshony nad...@gmail.com
wrote:

Thanks you for the response

when i insert data via ES and query it, all is fine.
the problem is when data is inserted to Couchbase..

Nadav

On Thursday, February 12, 2015 at 4:03:01 PM UTC+2, Itamar Syn-Hershko
wrote:

http://code972.com/blog/2015/02/80-elasticsearch-one-tip-a-d
ay-managing-index-mappings-like-a-pro
http://code972.com/blog/2015/02/81-elasticsearch-one-tip-a-d
ay-using-dynamic-templates-to-avoid-rigorous-mappings

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 3:59 PM, Nadav Hashimshony nad...@gmail.com
wrote:

Hi,

I'm new to the group, hope ill find what i need and share my
experience as i go along..

im using ES with the attachment-plugin in order to store and search
files.
when i set the mapping right and insert the file data in a Base64
manner I'm able to query my data via Kibana.

my problem is this.

if i create the index + mapping in ES, then insert the data to
Couchbase and use XDRC to replicate it to ES, i can't query the Data with
Kibana.
it looks like the mapping of the index created in ES doesn't index
well the data it gets from Couchbase.

has anyone encounter such an issue?

Thanks You

Nadav.

--
You received this message because you are subscribed to the Google
Groups elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/8092eaf5-0ef8-4249-8e5d-acff8281a81a%40goo
glegroups.com
https://groups.google.com/d/msgid/elasticsearch/8092eaf5-0ef8-4249-8e5d-acff8281a81a%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/1d9c8ce5-116f-40cc-a5e3-6ebe47191850%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/1d9c8ce5-116f-40cc-a5e3-6ebe47191850%40googlegroups.com?utm_medium=emailutm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8628ef07-2e10-423a-9de0-13ebaa37a0e8%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8628ef07-2e10-423a-9de0-13ebaa37a0e8%40googlegroups.com?utm_medium=emailutm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.

Re: ES OOMing and not triggering cache circuit breakers, using LocalManualCache

2015-02-12 Thread Wilfred Hughes

Oh, is field data per-node or total across the cluster? I grabbed a test 
cluster with two data nodes, and I deliberately set fielddata really low:

indices.fielddata.cache.size: 100mb

However, after a few queries, I'm seeing more than 100MiB in use:

$ curl http://localhost:9200/_cluster/stats?humanpretty;
...
  fielddata: {
memory_size: 119.7mb,
memory_size_in_bytes: 125543995,
evictions: 0
  },

Is this expected?

On Wednesday, 11 February 2015 18:57:28 UTC, Zachary Tong wrote:

 LocalManualCache is a component of Guava's LRU cache 
 https://code.google.com/p/guava-libraries/source/browse/guava-gwt/src-super/com/google/common/cache/super/com/google/common/cache/CacheBuilder.java,
  
 which is used by Elasticsearch for both the filter and field data cache. 
  Based on your node stats, I'd agree it is the field data usage which is 
 causing your OOMs.  CircuitBreaker helps prevent OOM, but it works on a 
 per-request basis.  It's possible for individual requests to pass the CB 
 because they use small subsets of fields, but over-time the set of fields 
 loaded into Field Data continues to grow and you'll OOM anyway.

 I would prefer to set a field data limit, rather than an expiration.  A 
 hard limit prevents OOM because you don't allow the cache to grow anymore. 
  An expiration does not guarantee that, since you could get a burst of 
 activity that still fills up the heap and OOMs before the expiration can 
 work.

 -Z

 On Wednesday, February 11, 2015 at 12:50:45 PM UTC-5, Wilfred Hughes wrote:

 After examining some other nodes that were using a lot of their heap, I 
 think this is actually field data cache:


 $ curl http://localhost:9200/_cluster/stats?humanpretty;
 ...
 fielddata: {
   memory_size: 21.3gb,
   memory_size_in_bytes: 22888612852,
   evictions: 0
 },
 filter_cache: {
   memory_size: 6.1gb,
   memory_size_in_bytes: 6650700423,
   evictions: 12214551
 },

 Since this is storing logstash data, I'm going to add the following lines 
 to my elasticsearch.yml and see if I observe a difference once deployed to 
 production.

 # Don't hold field data caches for more than a day, since data is
 # grouped by day and we quickly lose interest in historical data.
 indices.fielddata.cache.expire: 1d


 On Wednesday, 11 February 2015 16:29:22 UTC, Wilfred Hughes wrote:

 Hi all

 I have an ES 1.2.4 cluster which is occasionally running out of heap. I 
 have ES_HEAP_SIZE=31G and according to the heap dump generated, my biggest 
 memory users were:

 org.elasticsearch.common.cache.LocalCache$LocalManualCache 55%
 org.elasticsearch.indices.cache.filter.IndicesFilterCache 11%

 and nothing else used more than 1%.

 It's not clear to me what this cache is. I can't find any references to 
 ManualCache in the elasticsearch source code, and the docs: 
 http://www.elasticsearch.org/guide/en/elasticsearch/reference/1.x/index-modules-fielddata.html
  
 suggest to me that the circuit breakers should stop requests or reduce 
 cache usage rather that OOMing.

 At the moment my cache was filled up, the node was actually trying to 
 index some data:

 [2015-02-11 08:14:29,775][WARN ][index.translog   ] 
 [data-node-2] [logstash-2015.02.11][0] failed to flush shard on translog 
 threshold
 org.elasticsearch.index.engine.FlushFailedEngineException: 
 [logstash-2015.02.11][0] Flush failed
 at 
 org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:805)
 at 
 org.elasticsearch.index.shard.service.InternalIndexShard.flush(InternalIndexShard.java:604)
 at 
 org.elasticsearch.index.translog.TranslogService$TranslogBasedFlush$1.run(TranslogService.java:202)
 at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)
 Caused by: java.lang.IllegalStateException: this writer hit an 
 OutOfMemoryError; cannot commit
 at 
 org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4416)
 at 
 org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2989)
 at 
 org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3096)
 at 
 org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3063)
 at 
 org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:797)
 ... 5 more
 [2015-02-11 08:14:29,812][DEBUG][action.bulk  ] 
 [data-node-2] [logstash-2015.02.11][0] failed to execute bulk item (index) 
 index {[logstash-2015.02.11][syslog_slurm][1
 org.elasticsearch.index.engine.CreateFailedEngineException: 
 [logstash-2015.02.11][0] Create failed for 
 [syslog_slurm#12UUWk5mR_2A1FGP5W3_1g]
 at 
 org.elasticsearch.index.engine.internal.InternalEngine.create(InternalEngine.java:393)
 at

Re: Elasticsearch + attachment plugin + Kibana + couchbase

Yes, that too :)

Also if its a time based data, you will not be able to use kibana's date
filtering etc - because it lacks the @timestamp field. Basically, the XDCR
elasticsearch plugin was built around the XDCR / Couchbase realm and not
around Elasticsearch's. Unfortunately this means many ES features are
unavailable / hard to use, e.g.
https://github.com/couchbaselabs/elasticsearch-transport-couchbase/issues/63

https://github.com/couchbaselabs/elasticsearch-transport-couchbase/issues/64

I can help fixing this on the XDCR plugin if you'd like - ping me privately
and we can work something out (or I can convince you to avoid using the
XDCR replication)

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 5:18 PM, Nadav Hashimshony nad...@gmail.com wrote:

ok, ill try.

this envelope document, is it something i need to be concerned about
when I'm querying via Kibana?

On Thursday, February 12, 2015 at 5:14:39 PM UTC+2, Itamar Syn-Hershko
wrote:

Yes. Just make sure the template reflects the actual document structure -
as I said XDCR wraps your document in an envelope document

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 5:12 PM, Nadav Hashimshony nad...@gmail.com
wrote:

ok, just to be clear.

the steps i did was as followed:
1. create the index with the mapping.
2. define the XDCR to replicate my bucket with the index in ES.
3. insert data to couchbase.
4. try to query with kibana

What you suggest is to Add another BEFORE step 1:
0. create a template to include my mapping.
1. crate the index in ES
and so on...

did i get it right?

Thanks.
Nadav.

On Thursday, February 12, 2015 at 5:04:24 PM UTC+2, Itamar Syn-Hershko
wrote:

Like I said, you need the mapping to catch before the XDCR plugin
begins the replication - so you need to put a template with this mapping
that will override XDCR's

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 4:59 PM, Nadav Hashimshony nad...@gmail.com
wrote:

Thanks you for the response

when i insert data via ES and query it, all is fine.
the problem is when data is inserted to Couchbase..

Nadav

On Thursday, February 12, 2015 at 4:03:01 PM UTC+2, Itamar Syn-Hershko
wrote:

The XDCR plugin indexes the data using an envelope document. Long
story short, make sure you use the latest XDCR plugin as older ones are
missing lots of important functions, and use templates and dynamic
templates with proper field paths for this to work correctly

http://code972.com/blog/2015/02/80-elasticsearch-one-tip-a-d
ay-managing-index-mappings-like-a-pro
http://code972.com/blog/2015/02/81-elasticsearch-one-tip-a-d
ay-using-dynamic-templates-to-avoid-rigorous-mappings

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 3:59 PM, Nadav Hashimshony nad...@gmail.com
wrote:

Hi,

I'm new to the group, hope ill find what i need and share my
experience as i go along..

im using ES with the attachment-plugin in order to store and search
files.
when i set the mapping right and insert the file data in a Base64
manner I'm able to query my data via Kibana.

my problem is this.

if i create the index + mapping in ES, then insert the data to
Couchbase and use XDRC to replicate it to ES, i can't query the Data
with
Kibana.
it looks like the mapping of the index created in ES doesn't index
well the data it gets from Couchbase.

has anyone encounter such an issue?

Thanks You

Nadav.

--
You received this message because you are subscribed to the Google
Groups elasticsearch group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8092eaf5-0ef
8-4249-8e5d-acff8281a81a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8092eaf5-0ef8-4249-8e5d-acff8281a81a%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups elasticsearch group.
To unsubscribe from this group and stop receiving

Re: Elasticsearch + attachment plugin + Kibana + couchbase

Thanks you for the response

when i insert data via ES and query it, all is fine.
the problem is when data is inserted to Couchbase..

Nadav

On Thursday, February 12, 2015 at 4:03:01 PM UTC+2, Itamar Syn-Hershko
wrote:

http://code972.com/blog/2015/02/80-elasticsearch-one-tip-a-day-managing-index-mappings-like-a-pro

http://code972.com/blog/2015/02/81-elasticsearch-one-tip-a-day-using-dynamic-templates-to-avoid-rigorous-mappings

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 3:59 PM, Nadav Hashimshony nad...@gmail.com
javascript: wrote:

Hi,

I'm new to the group, hope ill find what i need and share my experience
as i go along..

im using ES with the attachment-plugin in order to store and search files.
when i set the mapping right and insert the file data in a Base64 manner
I'm able to query my data via Kibana.

my problem is this.

has anyone encounter such an issue?

Thanks You

Nadav.

https://groups.google.com/d/msgid/elasticsearch/8092eaf5-0ef8-4249-8e5d-acff8281a81a%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1d9c8ce5-116f-40cc-a5e3-6ebe47191850%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Moving to production - Storage and servers

Hi everyone,

After my first 
https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/US-BA4R_Qdc 
and second question 
https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/FGKUmzn-WSs 
regarding clusters' examples and multy-tenancy with garantee of service, I 
have one more on my way to production :

Could you describe the *pros* and *cons* of the :
   
   1. Disks : *DAS* (Direct Attached Storage) - *JBOD* (Just a Bunch Of 
  Disks) / *SAN* (Storage Area Network)? I take it that NAS is most of 
  the time to be avoided
  2. Servers : bare metal servers / virtual servers ?
   
For those interested :

   - JBOD / SAN storage discussion in To Raid or not to Raid:
   
   
https://groups.google.com/forum/?fromgroups#!searchin/elasticsearch/hardware/elasticsearch/HSj2fZGdU1Y/4mFCBTCb-JcJ
   - Doc which states about SSDs, RAID0, NAS, networks
   
   
http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/hardware.html

Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/3ab36482-e4fb-4dd0-99cd-044d2e4257ea%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch + attachment plugin + Kibana + couchbase

ok, ill try.

this envelope document, is it something i need to be concerned about when
I'm querying via Kibana?

On Thursday, February 12, 2015 at 5:14:39 PM UTC+2, Itamar Syn-Hershko
wrote:

Yes. Just make sure the template reflects the actual document structure -
as I said XDCR wraps your document in an envelope document

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 5:12 PM, Nadav Hashimshony nad...@gmail.com
javascript: wrote:

ok, just to be clear.

the steps i did was as followed:
1. create the index with the mapping.
2. define the XDCR to replicate my bucket with the index in ES.
3. insert data to couchbase.
4. try to query with kibana

What you suggest is to Add another BEFORE step 1:
0. create a template to include my mapping.
1. crate the index in ES
and so on...

did i get it right?

Thanks.
Nadav.

On Thursday, February 12, 2015 at 5:04:24 PM UTC+2, Itamar Syn-Hershko
wrote:

Like I said, you need the mapping to catch before the XDCR plugin
begins the replication - so you need to put a template with this mapping
that will override XDCR's

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 4:59 PM, Nadav Hashimshony nad...@gmail.com
wrote:

Thanks you for the response

when i insert data via ES and query it, all is fine.
the problem is when data is inserted to Couchbase..

Nadav

On Thursday, February 12, 2015 at 4:03:01 PM UTC+2, Itamar Syn-Hershko
wrote:

The XDCR plugin indexes the data using an envelope document. Long
story short, make sure you use the latest XDCR plugin as older ones are
missing lots of important functions, and use templates and dynamic
templates with proper field paths for this to work correctly

http://code972.com/blog/2015/02/80-elasticsearch-one-tip-a-d
ay-managing-index-mappings-like-a-pro
http://code972.com/blog/2015/02/81-elasticsearch-one-tip-a-d
ay-using-dynamic-templates-to-avoid-rigorous-mappings

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 3:59 PM, Nadav Hashimshony nad...@gmail.com
wrote:

Hi,

I'm new to the group, hope ill find what i need and share my
experience as i go along..

im using ES with the attachment-plugin in order to store and search
files.
when i set the mapping right and insert the file data in a Base64
manner I'm able to query my data via Kibana.

my problem is this.

if i create the index + mapping in ES, then insert the data to
Couchbase and use XDRC to replicate it to ES, i can't query the Data
with
Kibana.
it looks like the mapping of the index created in ES doesn't index
well the data it gets from Couchbase.

has anyone encounter such an issue?

Thanks You

Nadav.

--
You received this message because you are subscribed to the Google
Groups elasticsearch group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/8092eaf5-0ef8-4249-8e5d-acff8281a81a%40goo
glegroups.com
https://groups.google.com/d/msgid/elasticsearch/8092eaf5-0ef8-4249-8e5d-acff8281a81a%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/1d9c8ce5-116f-40cc-a5e3-6ebe47191850%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/1d9c8ce5-116f-40cc-a5e3-6ebe47191850%40googlegroups.com?utm_medium=emailutm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

Re: A strange behavior we've encountered on our ELK