Re: issue with elastic search TransportClient of java API

2014-09-26 Thread David Pilato
How your Java code looks like? What was your curl query? -- David ;-) Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 26 sept. 2014 à 07:34, Vijay Tiwary vijaykr.tiw...@gmail.com a écrit : I am having a instance of TransportClient which is singleton in my web application. In one

Re: issue with elastic search TransportClient of java API

2014-09-26 Thread Vijay Tiwary
Blow is the junit test class: public class BenchMarkES { private static final Logger LOG = LoggerFactory.getLogger(BenchMarkES.class); private static TransportClient client = null; @Before public void setUp() { Settings settings = ImmutableSettings.settingsBuilder().build(); client = new

Re: issue with elastic search TransportClient of java API

2014-09-26 Thread David Pilato
Just checking. Which version you elasticsearch cluster is? -- David ;-) Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 26 sept. 2014 à 08:28, Vijay Tiwary vijaykr.tiw...@gmail.com a écrit : Blow is the junit test class: public class BenchMarkES { private static

Re: Adding a field dinamically to search results

2014-09-26 Thread Dani Castro
Nobody has faced something like this before? :( El jueves, 25 de septiembre de 2014 18:33:08 UTC+2, Dani Castro escribió: Hi, I am a newbie into ElasticSearch and I am trying to figure out how to achieve this: In our Elastic Search Cluster we have documents like this: { hotel :

Any idea to remove the duplicates from the search results?

2014-09-26 Thread Peiyong Lin
Hi all, I have some documents that look like: { name: , image_url: , simHash: } And there are many duplicate documents with the same `simHash` value, any idea to remove the duplicates instead of removing them in indexing time? I have seen solutions here:

Re: Elasticsearch threads behaves different from each other

2014-09-26 Thread Umut Yerci
Thank you for answers Jörg. I really appreciated your helps :-). On Thursday, September 25, 2014 7:07:51 PM UTC+3, Jörg Prante wrote: What you see are some of the TCP/IP stack counters of your hardware network interface since it was started. It is provided by the OS and read by sigar

Re: Loading JSON-LD into ES

2014-09-26 Thread joergpra...@gmail.com
JSON-LD is perfect for ES indexing, as long as you use the compact form of representation. http://www.w3.org/TR/json-ld-api/#compaction-algorithms Example: https://github.com/lanthaler/JsonLD/blob/master/Test/Fixtures/sample-compacted.jsonld This means you should use short field names and

Re: issue with elastic search TransportClient of java API

2014-09-26 Thread Vijay Tiwary
Hi David, I have identified the problem. Actually the transport client that i was creating was like this: Settings settings = ImmutableSettings.settingsBuilder().put(client.transport.sniff, true).build(); client = new TransportClient(settings).addTransportAddress(new

Re: Transport Client hangs in my web application during search.

2014-09-26 Thread Vijay Tiwary
If you turn off your sniff settings to false it will work. On Wednesday, January 8, 2014 2:11:10 AM UTC+5:30, Search User wrote: I have a web application in which I create a Transport Client using Spring (singleton) and inject it into my service. When I receive a request in my controller,

issue with elastic search river plugin

2014-09-26 Thread kexu xu
Hi,all when load data to elastic search using river, I always get the error: ClassNotFoundException. The fellow is the error description: The elastic search has 3 nodes,their ip is 192.168.71.234,192.168.71.235 and 192.168.71.236 First, Install the river plugin, the 3 node return right,

Re: Loading JSON-LD into ES

2014-09-26 Thread Lukáš Vlček
Jörg, my concern is that RDF/XML allow to express one thing in several ways. For example, if you take FOAF specification then there are several ways how you can express that one Person knows other Person. One way it using reference IDs other way it using nested Person inside other Person. See [1]

RE: Any idea to remove the duplicates from the search results?

2014-09-26 Thread Tugberk Ugurlu
Hi, You are probably looking for the same thing that I was looking for a few days ago: https://groups.google.com/forum/#!searchin/elasticsearch/tugberk/elasticsearch/1uCQ7R8vCS8/-iRJLrdGGrYJ Top hits aggregation

Re: Significant terms aggregation with non tokenized text

2014-09-26 Thread Mark Harwood
Unlike the terms aggs which only accesses the content loaded into RAM (aka FieldData), the significant_terms agg has to also go to disk to check the frequency of terms in the index for the background count. This different datasource means the naming conventions can sometimes differ. Can you try

Re: More efficient date sorting

2014-09-26 Thread Anantha Govindarajan
Hi, Initially we too used System.currentTimeInMillis() . Then we switched to 2 int fields something like MMdd HHmmssSSS. If query's time criteria falls with in single date then we dont apply *MMdd field* for sorting. We have decent performance compared with

More like these, example query wanted

2014-09-26 Thread Ola Gustafsson
The documentation says More Like This can find documents that are like a set of chosen documents. The syntax to specify one or more documents is similar to the Multi GET API, and supports the ids or docs array.

How can I define a Username and password for login?

2014-09-26 Thread dalghak . joker
Hello Members. How are you? I use Elastecsearch+Kibana+Nginx+Logstach for record windows Logs and I want to know how can I define an account for browse Kibana? by default any user that enter my Log server address in Browser, Can see my Log server. Thank you and Good luck. -- You received

Re: How can I define a Username and password for login?

2014-09-26 Thread vineeth mohan
Hello , ELK stack doesn't give inherent support for this , but then there are a couple of open source projects you can use. https://github.com/fangli/kibana-authentication-proxy works fine for me. Or else a solution based on Nginx will also work good. Thanks Vineeth On Fri, Sep 26,

Re: Loading JSON-LD into ES

2014-09-26 Thread joergpra...@gmail.com
Lukáš, of course you are right, RDF/XML looks complex and requires parsing. The underlying principle of all RDF is a graph (or a series of triples in form of subject/predicate/object, where the triple series is a serialization of the graph), So the challenge is first the parsing of RDF input, and

Re: Kibana server-size integration with R, Perl, and other tools

2014-09-26 Thread Ashit Kumar
Brian, I agree completely with your expectations. If I am to replace Splunk (ridiculously overpriced in my opinion) or Logrhythm, I need to be able to : 1. Generate alerts that need immediate reaction. 2. Generate reports - Compliance related reporting - Perform aggregations on the fly

Re: More efficient date sorting

2014-09-26 Thread joergpra...@gmail.com
Your formula is not correct. yyyMMdd would map all values of a day to a single integer and you get something like sort by day or filter by day. Assumed you have a normal distribution and you have a year of timestamps, you can estimate: 80mio / 365 = 219.178 timestamps per day. In the day field,

TransportClientNodesService RetryListener.onFailure stackoverflow error

2014-09-26 Thread Jason Wee
Hi Elasticsearch, https://github.com/elasticsearch/elasticsearch/blob/v0.90.7/src/main/java/org/elasticsearch/client/transport/TransportClientNodesService.java#L262 Will that line cause java.lang.StackOverflowError? If yes, is there a way to mitigate the problem? Thank you. Jason -- You

Fuzziness AUTO doesn't work as expected

2014-09-26 Thread David Klotz
Hi, I'm currently having some issues with a search that's using Fuzziness.AUTO. First a remark on how AUTO is documented, the documentation says: For lengths: 0..1 must match exactly, 1..4 one edit allowed, 4 two edits allowed. That is a bit ambigue, how is a term with length 1 treated? I'm

Re: Loading JSON-LD into ES

2014-09-26 Thread Alfredo Serafini
Hi using json-ld is indeed rather simple, as it is JSON, and then it's even possible to index it as is. I'm currently using ES for storing RDF documents in json-ld on a specific index: in that case one can simply use the uri as an _id, recover the full original format by _source, and use

Re: issue with elastic search river plugin

2014-09-26 Thread David Pilato
Did you restart your nodes? --  David Pilato | Technical Advocate | elasticsearch.com david.pil...@elasticsearch.com @dadoonet | @elasticsearchfr | @scrutmydocs Le 26 septembre 2014 à 09:43:10, kexu xu (xuke...@gmail.com) a écrit:  Hi,all    when load data to elastic search using river, I

Re: issue with elastic search TransportClient of java API

2014-09-26 Thread David Pilato
I have no idea. Could be an issue. Any chance you could create a small test project which reproduce it and share it on github? --  David Pilato | Technical Advocate | elasticsearch.com david.pil...@elasticsearch.com @dadoonet | @elasticsearchfr | @scrutmydocs Le 26 septembre 2014 à

Re: More efficient date sorting

2014-09-26 Thread Anantha Govindarajan
Hi Jörg, Sorry I intended to say for HHmmssSSS field . How much memory will es take when I apply sorting or aggregations on HHmmssSSS field . In this case number of unique values for HHmmssSSS field can be 8640(~80.6 million.

Logtsash output to ElasticSearch Cluster

2014-09-26 Thread Kevin M
Do i need to set anything up special to have logstash output to my elasticsearch cluster? output { stdout { } elasticsearch { cluster = es-cluster } } I have my logstash output setup this way but I do not get anything in kibana - when I switch to elasticsearch_http it works fine

Re: Loading JSON-LD into ES

2014-09-26 Thread joergpra...@gmail.com
Absolutely. My thought is about managing one (or more) context ES JSON document(s) where all the @context definitions of an index live. A format plugin can then process search results and converts ES JSON to expanded JSON-LD and from there to other RDF serializations. Jörg On Fri, Sep 26, 2014

Re: msgpack support

2014-09-26 Thread Otis Gospodnetic
Hi, I don't know of it, although I see https://github.com/elasticsearch/logstash/blob/master/lib/logstash/codecs/msgpack.rb Did you maybe implement this already? Thanks, Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support *

Re: Kibana server-size integration with R, Perl, and other tools

2014-09-26 Thread Lance A. Brown
On 2014-09-25 11:57 am, Brian wrote: And as my part of the bargain, I will use Perl, R, or whatever else is at my disposal to create custom commands that can run on the Kibana host and perform all of the analysis that our group needs. Something to remember: The Kibana host is your browser.

Re: issue with elastic search TransportClient of java API

2014-09-26 Thread Ivan Brusic
In general, newer client libraries should not be used with older clusters. Most of the version checking happens on the server side and the older code does not know about the newer client. -- Ivan On Fri, Sep 26, 2014 at 9:54 AM, David Pilato da...@pilato.fr wrote: I have no idea. Could be an

Re: issue with elastic search TransportClient of java API

2014-09-26 Thread David Pilato
Yeah. Still that's super strange it works for a single call. -- David ;-) Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 26 sept. 2014 à 22:49, Ivan Brusic i...@brusic.com a écrit : In general, newer client libraries should not be used with older clusters. Most of the version

Upper bounds on the number of indexes in an elastic search cluster

2014-09-26 Thread Todd Nine
Hey guys. We’re building a Multi tenant application, where users create applications within our single server.For our current ES scheme, we're building an index per application. Are there any stress tests or documentation on the upper bounds of the number of indexes a cluster can

Re: Upper bounds on the number of indexes in an elastic search cluster

2014-09-26 Thread joergpra...@gmail.com
Why do you want to create huge number of indexes on just a single node? There are smarter methods to scale. Use over-allocation of shards. This is explained by kimchy in this thread http://elasticsearch-users.115913.n3.nabble.com/Over-allocation-of-shards-td3673978.html TL;DR you can create

Re: Upper bounds on the number of indexes in an elastic search cluster

2014-09-26 Thread Todd Nine
Hi Jorg, We're storing each application in it's own Index so we can manage it independently of others. There's not set load or usage on our applications. Some will be very small, a few hundred documents. Others will be quite large, in the billions. We have no way of knowing what the

Re: Upper bounds on the number of indexes in an elastic search cluster

2014-09-26 Thread joergpra...@gmail.com
If you consider tens of thousands of indices on tens of thousands of nodes, and the master node is the only node that can write to the cluster state, it will have lot of work to do to keep up with all cluster state updates. When the rate of changes to the cluster state increases, the master node

Re: Upper bounds on the number of indexes in an elastic search cluster

2014-09-26 Thread Todd Nine
It sounds like we're going to need to test our upper bounds of indexes (with no data) to see how many we can support. We may need to re-evaluate our thoughts on an index per app. We might be better off doing a statically sized set of indexes, then consistently hashing our applications to those

Re: Replace failing disks on a single node

2014-09-26 Thread David Pilato
Is your cluster still yellow? It should be Green at some point unless you change some settings explicitly. If your cluster does not index anymore, you could copy manually files in data dir and copy them on your new disk. But I wonder how you can copy from a failing disk? I'd probably let

Re: How can I define a Username and password for login?

2014-09-26 Thread chenlin rao
I implement a kbnauth in my `https://github.com/chenryn/kibana`. May anyone give a try? 2014-09-26 19:49 GMT+08:00 vineeth mohan vm.vineethmo...@gmail.com: Hello , ELK stack doesn't give inherent support for this , but then there are a couple of open source projects you can use.

Re: Equivalent aggregation query to terms facet

2014-09-26 Thread Abhishek Andhavarapu
I think I figured it out. Each aggs sections sub divides the data. First aggs buckets all the nested docs. Second aggs restricts the bucket to a filter. Third aggs run the terms facet on it. aggs: { foo: { nested: { path: rdata }, aggs: { bar: {