[ANN] Elastic4s Scala DSL 1.0.0 now available
Hi all, Pleased to announce release 1.0.0 of elastic4s. https://github.com/sksamuel/elastic4s Available on maven central. Thanks to everyone who contributed to this release. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/23ec8aed-020b-40cb-aeed-7332917697b4%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Unicast discovery fails to connect to master
Not sure about your architecture. May be you have good reasons for that but
running more than one node per machine is not what I'd recommend.
But here, may be they are client only nodes?
Specifying port range is OK. When your node starts, it tries to ping b001:9300,
then b001:9301, …
One node should at least answer otherwise, current node will think it's alone
and will set itself as master.
Once the cluster state is get from master node to the current node, current
node knows exactly which nodes forms the cluster and on which port for each
node.
So, when pinging, only the right port is pinged. If the expected node does not
answer to ping request, it will be considered as leaving the cluster.
Makes sense?
--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr
Le 14 février 2014 à 08:51:47, Robin Verlangen (ro...@us2.nl) a écrit:
Hi David,
Ok, I think the latter (the port range) is the actual issue. Would you be able
to think with me on a possible solution?
We run 3 ES applications per node, each with 2-4 ES clients. Would it then be
best to specify b001.my-cluster.com[9300-9312],b002.my-cluster.com[9300-9312]
et cetera?
Best regards,
Robin Verlangen
Chief Data Architect
W http://www.robinverlangen.nl
E ro...@us2.nl
What is CloudPelican?
Disclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that the
information remains the property of the sender. You must not use, disclose,
distribute, copy, print or rely on this e-mail. If you have received this
message in error, please contact the sender immediately and irrevocably delete
this message and any copies.
On Fri, Feb 14, 2014 at 8:46 AM, David Pilato da...@pilato.fr wrote:
Ping will ping one port.
If you did not set port in unicast list, 9300 is assumed I guess.
Modify elasticsearch.yml file and set the right port for this node.
HTH
--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
Le 14 févr. 2014 à 08:36, Robin Verlangen ro...@us2.nl a écrit :
In addition to my previous question, is it correct that version 0.90.11 only
pings one port, instead of the entire 9300-9400 range?
https://github.com/elasticsearch/elasticsearch/blob/a4b2366e1e50953b7308b21963133dc50dd3fc60/src/main/java/org/elasticsearch/discovery/zen/ping/unicast/UnicastZenPing.java#L112
Best regards,
Robin Verlangen
Chief Data Architect
W http://www.robinverlangen.nl
E ro...@us2.nl
What is CloudPelican?
Disclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that the
information remains the property of the sender. You must not use, disclose,
distribute, copy, print or rely on this e-mail. If you have received this
message in error, please contact the sender immediately and irrevocably delete
this message and any copies.
On Fri, Feb 14, 2014 at 8:30 AM, Robin Verlangen ro...@us2.nl wrote:
Hi there,
We're having issues with a cluster that fails to connect to it's master
repeatedly. Please see the logs below:
INFO: [b002.my-cluster.com] failed to send join request to master
[[b005.my-cluster.com][Hpm2Z7AaR3ugg417majMQg][inet[/37.139.25.xxx:9302]]],
reason [org.elasticsearch.ElasticSearchTimeoutException: Timeout waiting for
task.]
Feb 14, 2014 7:23:47 AM org.elasticsearch.discovery.zen.ping.unicast
WARNING: [b002.my-cluster.com] failed to send ping to
[[#zen_unicast_1#][inet[b001.my-cluster.com/85.17.231.xxx:9300]]]
org.elasticsearch.transport.ReceiveTimeoutTransportException:
[][inet[b001.my-cluster.com/85.17.231.xxx:9300]][discovery/zen/unicast]
request_id [18] timed out after [3750ms]
at
org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:356)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Feb 14, 2014 7:23:49 AM org.elasticsearch.discovery.zen
INFO: [b002.my-cluster.com] master_left
[[b005.my-cluster.com][Hpm2Z7AaR3ugg417majMQg][inet[/37.139.25.xxx:9302]]],
reason [do not exists on master, act as master failure]
Feb 14, 2014 7:23:49 AM org.elasticsearch.discovery
INFO: [b002.my-cluster.com] my-cluster-001/Vjs0tUn7QTq8oDN2F0PxQQ
Feb 14, 2014 7:23:49 AM org.elasticsearch.http
INFO: [b002.my-cluster.com] bound_address {inet[/0:0:0:0:0:0:0:0:9200]},
publish_address {inet[/37.139.5.xxx:9200]}
Feb 14, 2014 7:23:49 AM org.elasticsearch.node
INFO: [b002.my-cluster.com] started
Best regards,
Robin Verlangen
Chief Data Architect
W http://www.robinverlangen.nl
E ro...@us2.nl
What is CloudPelican?
Disclaimer: The information contained in this message and attachments
Re: Marvel not collecting system metrics
How did you install Marvel? You need to add Marvel plugin on every node and each node must be restarted. Also take care of the setting marvel.agent.exporter.es.hosts if you use other host/port than localhost:9200 Jörg -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEWrMoZPm8bHfkQbq_-uJojAFLQgaexu0UXXjz5M%2B0biQ%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Marvel not collecting system metrics
Hi, As I am using Docker Elasticsearch is not running during the build process. I have posted my Dockerfile below. https://github.com/damm/dockerfiles/blob/master/elasticsearch/Dockerfile#L7 FWIW my /_node/stats is correct and fully populated. Additionally I have looked over https://groups.google.com/forum/#!msg/elasticsearch/UPBMKvxwwM8/VBaqh0B11tQJand tried applying that as a custom template and deleting my indices and starting over; did not help. Additionally the elasticsearch.yml in that repo is the current one in use. Thanks! On Fri, Feb 14, 2014 at 12:17 AM, joergpra...@gmail.com joergpra...@gmail.com wrote: How did you install Marvel? You need to add Marvel plugin on every node and each node must be restarted. Also take care of the setting marvel.agent.exporter.es.hosts if you use other host/port than localhost:9200 Jörg -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/dhpxaOuoZWI/unsubscribe. To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEWrMoZPm8bHfkQbq_-uJojAFLQgaexu0UXXjz5M%2B0biQ%40mail.gmail.com . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALDJu4XJRFyWWk7mhHcKt2i%3D9-kXf8Ji54LQTLU4ebmAjTZNRA%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Getting started on a Mac (Splunk user looking to migrate)
Please, I'd like to forward you to the nice Elasticsearch company to ask them if they can provide the service you request, for example, something like a native Mac OS X dmg package of the ELK stack, with OOTB experience. I'm quite sure they have something like this in the pipeline, because expectations are getting higher and higher. The community mailinglist can help if you have questions about Elasticsearch. Elasticsearch is not a Splunk replacement but a search engine that can power such platforms. Many apps can be added to ES, and Splunk-like apps are just one possibility. Don't be annoyed because you just tried to install the backbone of a whole platform. If your question 2a) about MySQL is separate from this, you could try the JDBC river plugin I wrote. https://github.com/jprante/elasticsearch-river-jdbc It is built to pull quickly DB result sets out of any JDBC source with SQL statements and index JSON documents into Elasticsearch, without deeper knowledge of programming JDBC or ES. It's ideal to get some feet wet for PoCs and live showcases. It's not a full-fledged solution for every use case though. You are free to adapt the JDBC river to your needs, it's open source. There are other river plugins or solutions available for question 2b) and 2c) like for example http://www.scrutmydocs.org/ As you have noticed, there is no pricing info about Elasticsearch. It's source code is licensed with Apache License 2.0 and it is free to download, compile, modify, and distribute. That means, you can build your business around it, build dashboards as you like etc. Ask the company for service, I'm sure they can help you. Jörg On Fri, Feb 14, 2014 at 2:52 AM, Phoenix Kiula phoenix.ki...@gmail.comwrote: Hi. I'm somewhat familiar with Splunk, because our company invested in it. Now it's proving to be expensive so we're looking for alternatives. (It's a super tool though) ElasticSearch was recommended on Quora. But downloading and trying to install it is hideous. This seems for tech nerds. Or am I missing something? I'm on a Mac OSX and want to try it out. The Splunk or SumoLogic installs were 10 seconds and I was up and running. Could someone please point me to a simple guide to: 1. Install the whole thing: including LogStash and Kibana, because just searching is not what I wish to do, I want to make sense of the data and therefore dashboarding is important. I couldn't find a simple step by step tutorial or guide on the website to install this whole stack. Google turns up stuff like this -- http://red-badger.com/blog/2013/11/08/getting-started-with-elasticsearch/ -- which is ridiculously dysfunctional. (And I don't want to install brew just to install simple stuff.) 2. Secondly, once installed, I'd like to know how to simply get the data into the index from: a) a MySQL database b) a folder with XML files...I want to slurp all the files into the index, and then incrementally only get new files in the future c) a folder with text log files...and same incremental auto-pickup as (b) above Don't seem to find a simple non-geeky guide to do this. I'm a programmer in PHP and web technologies (JS etc), and manage my own dedicated Linux hosting, so not averse to code or commands, but want to find some coherent and simple guide. 3. Thirdly, the demo.kibana.org is hideous. It looks like the best way to UN-sell this technology. Is there a better place I could see actual Kibana in use, you know, with a proper dashboard that doesn't look like a screen from The Matrix in dark black? I'm looking for a business dashboard with proper dropdown based filters etc. And the ability perhaps to use third party libraries such as D3 JS. Finally, where's the pricing info? Is ElasticSearch completely free? There's no pricing info anywhere, only the cost of helping out. Is this why the documentation etc is so poor so that they can charge for helping? Many thanks for any pointers. I really want to give this a shot. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/10b30c8e-2554-437d-a48c-39ed38b4f0d3%40googlegroups.com . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFykrRWfWEPeoq%2BcuN0iyPrEyhOt7HuafNOPmFgcRUOhA%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Filter facet by term count
This is typically something that you can do using a terms aggregation[1].
It would look something like:
{
aggs : {
top_ips : {
terms : {
field : ip_address, // - change field name accordingly
min_doc_count: 100
}
}
}
}
[1]
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html
On Fri, Feb 14, 2014 at 7:11 AM, Anton T at1650...@gmail.com wrote:
Hello, I have an apache log in ES and I want to get top daily IPs having
hit count 100, for example. Tried to use scripting but i have no idea how
to get to 'count' value of a term. Any advice?
Thank you.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/e80ef007-1897-4a03-b5d9-79ce123af8d6%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.
--
Adrien Grand
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j79draJz5HTV6%2B3iWWj56KdwoDiEX%3Ds9faXT%2Bqe1HLXYg%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: Issue with unit testing TransportClient
Hey,
you might want to read:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/testing-framework.html
should help you to create tests which have a whole elasticsearch cluster
running in the background.
--Alex
On Thu, Feb 13, 2014 at 7:37 PM, joergpra...@gmail.com
joergpra...@gmail.com wrote:
I admit the code is confusing and too short, sorry.
What you want is just exactly what I do.
I start a test elasticsearch test cluster by doing this
@BeforeMethod
public void create() throws Exception {
startNode(1);
NodesInfoRequest nodesInfoRequest = new
NodesInfoRequest().transport(true);
NodesInfoResponse response =
client(1).admin().cluster().nodesInfo(nodesInfoRequest).actionGet();
InetSocketTransportAddress address =
(InetSocketTransportAddress)response.iterator().next()
.getTransport().getAddress().publishAddress();
PORT = address.address().getPort();
addresses.put(1, address);
}
...
protected URI getAddress() {
return URI.create(es://localhost: + PORT + ?es.cluster.name=
+ CLUSTER);
}
and now I can tell a TransportClient to use the server. For this I keep a
map of adresses of the server nodes in form of an URI.
Look at this code, which is trivially extending the abstract test code
@Test
public void testBulkClient() {
final BulkClient es = new BulkClient()
.newClient(getAddress());
es.shutdown();
if (es.hasErrors()) {
logger.error(error, es.getThrowable());
}
assertFalse(es.hasErrors());
}
I use getAddress() for the address of the elasticseach test cluster that
is currently running, and the BulkClient (which is my custom client
wrapping a TransportClient together with a BulkProcessor) is connecting to
the embedded test cluster with newClient(...). In this method I build a
Settings object that contains all the required info for the TransportClient
to connect.
The Setting object is something like
settingsBuilder()
.put(cluster.name, findClusterName(uri))
.put(network.server, false)
.put(node.client, true)
.put(client.transport.sniff, false)
.put(client.transport.ignore_cluster_name, false)
.put(client.transport.ping_timeout, 30s)
.put(client.transport.nodes_sampler_interval, 30s)
.build();
where findClusterName() is a routine to get the cluster name from the
parameters of the test.
The @AfterMethod shuts everything down, the BulkClient and the test
cluster.
If i want more nodes, I add startNode(2), startNode(3), to the
@BeforeMethod
Does this help?
Jörg
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEL2w5kJby8PddkvHRdZNtAima3E1vAaEYOW%3D3mQN2_og%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGCwEM9J7g23LTiDwci6xu5wcTKyTMxpsCancLLdjdKteur6tw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: Elasticsearch global search different filter on multiple indexes..
Thanks for the prompt response. Aliases will surely help me in resolving filter issue across indexes but I didn't get how would I exclude certain fields globally from highlightling. Our requirement is to highlight all the fields except few fields. On Thursday, 13 February 2014 19:49:10 UTC+5:30, Binh Ly wrote: It is possible to define multiple aliases (each with a different filter condition) and then search across aliases. Might help in your case. http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-aliases.html About highlighting, if you mean can you selectively highlight on specific fields, yes you can supply a list of fields that ES will perform the highlighting on. More details here: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-highlighting.html -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a4f9db18-c12d-42cc-90ca-1bd1e0631e6c%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Extending Thai analyzer.
Hey,
the standard thai analyzer supports a stopwords_path in the mapping, no
need to reference to that ThaiWordFilterFactory...
Should help you.
--Alex
On Fri, Feb 14, 2014 at 3:06 AM, Min Cha minslo...@gmail.com wrote:
Hello Nik.
Thanks for your advice.
I had just tried as you advice. But, I met an error as following.
error: IndexCreationException[[search] failed to create index]; nested:
CreationException[Guice creation errors:\n\n1) Could not find a suitable
constructor in org.apache.lucene.analysis.th.ThaiWordFilterFactory. Classes
must have either one (and only one) constructor annotated with @Inject or a
zero-argument constructor that is not private.\n at
org.apache.lucene.analysis.th.ThaiWordFilterFactory.class(Unknown Source)\n
at
org.elasticsearch.index.analysis.TokenFilterFactoryFactory.create(Unknown
Source)\n at
org.elasticsearch.common.inject.assistedinject.FactoryProvider2.initialize(Unknown
Source)\n at _unknown_\n\n1 error]; ,
In my opnion, this error raises by ThaiWordFilterFactory which has`t a
zeo-argument constructor. In fact, the ThaiWordFilterFactory has only a
following constructor.
/** Creates a new ThaiWordFilterFactory */
public ThaiWordFilterFactory(MapString,String args) {
super(args);
assureMatchVersion();
if (!args.isEmpty()) {
throw new IllegalArgumentException(Unknown parameters: + args);
}
}
If you don`t mind, I have an one more question. Can I define a constructor
argument in above settings JSON.
2014년 2월 7일 금요일 오후 11시 17분 59초 UTC+9, Nikolas Everett 님의 말:
If you don't like the language analyzer you have to rebuild it as a
custom analyzer then add what you need to it.
{
analyzer: {
thai_with_ngram: {
type: custom,
tokenizer: standard,
filters: [standard, lowercase, thai, thai_stop, ngram]
}
},
filter: {
thai: {
type: org.apache.lucene.analysis.th.ThaiWordFilterFactory
},
thai_stop: {
type: stop,
stopwords_path: org/apache/lucene/analysis/th/stopwords.txt
},
ngram: { your ngram configuration here }
}
}
Builds it with your ngram configuration. I think. I'm taking quite a
few educated guesses here so I expect you to have to fiddle with it to get
it right.
How I did this:
1. Open the class called ThaiAnalyzer in the Lucene version
Elasticsearch is using and find the method called createComponents. For me
this is simple because I have Elasticsearch open in Eclipse.
2. That method defines the tokenizer (standard) and some filters
(standard, lowercase, ThaiWordFilter, and stop. You have to be able to
translate the class names to Elasticsearch's easier names to get this to
work properly.
3. Now build it as a custom filter with your extra filter in there.
That is thai_with_ngram above.
4. Next you'll need to define all the filters that don't exist by
default in Elasticsearch. In this case that is thai, thai_stop, and your
ngram filter. In order:
5. The thai filter doesn't have an easy Elasticsearch mapping so you
have to tell Elasticsearch the class name to load. That class doesn't take
an configuration so we're done.
6. The thai_stop filter is just a regular stop word filter with thai
stop words. But Elasticserach doesn't have an easy name to reference the
thai stop words file. That isn't too bad, as you can load the stopwords
file from the classepath. It lives in Lucene at the path I added above.
7. The ngram filter is yours to build but it is well documented.
That took longer then I expected but it was worth the exercise so I'll
remember how to do it again when I need it. For reference, I do it for
English which has more filters but they all have easy names.
Nik
On Fri, Feb 7, 2014 at 12:59 AM, Min Cha mins...@gmail.com wrote:
Hi folks.
I would like to develop for a searching system for Thai language.
First of all, I found Thai analyzer and it seemed like good.
Actually, but, It doesn`t meet my whole requirement.
I decided what extends it.
For example, I would like to add nGram token filter on the Thai analyzer
without any changes on it.
How to do this?
Please, give me some advice.
--
You received this message because you are subscribed to the Google
Groups elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/5041f397-8732-413f-8e50-46e25610c639%
40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
Re: Filter facet by term count
Wow. Have to upgrade ES from 0.9 to 1.x.
Thank you.
On Friday, February 14, 2014 1:08:30 PM UTC+4, Adrien Grand wrote:
This is typically something that you can do using a terms aggregation[1].
It would look something like:
{
aggs : {
top_ips : {
terms : {
field : ip_address, // - change field name accordingly
min_doc_count: 100
}
}
}
}
[1]
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html
On Fri, Feb 14, 2014 at 7:11 AM, Anton T at16...@gmail.com
javascript:wrote:
Hello, I have an apache log in ES and I want to get top daily IPs having
hit count 100, for example. Tried to use scripting but i have no idea how
to get to 'count' value of a term. Any advice?
Thank you.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com javascript:.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/e80ef007-1897-4a03-b5d9-79ce123af8d6%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.
--
Adrien Grand
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/e55b75fe-8fc2-4478-ba7d-007cf471691d%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: Extending Thai analyzer.
Thanks.
If you dont mind, can you give me a specific example or explain more
specific?
I cant`t understand your advice.
2014년 2월 14일 금요일 오후 6시 55분 44초 UTC+9, Alexander Reelsen 님의 말:
Hey,
the standard thai analyzer supports a stopwords_path in the mapping, no
need to reference to that ThaiWordFilterFactory...
Should help you.
--Alex
On Fri, Feb 14, 2014 at 3:06 AM, Min Cha mins...@gmail.com
javascript:wrote:
Hello Nik.
Thanks for your advice.
I had just tried as you advice. But, I met an error as following.
error: IndexCreationException[[search] failed to create index];
nested: CreationException[Guice creation errors:\n\n1) Could not find a
suitable constructor in
org.apache.lucene.analysis.th.ThaiWordFilterFactory. Classes must have
either one (and only one) constructor annotated with @Inject or a
zero-argument constructor that is not private.\n at
org.apache.lucene.analysis.th.ThaiWordFilterFactory.class(Unknown Source)\n
at
org.elasticsearch.index.analysis.TokenFilterFactoryFactory.create(Unknown
Source)\n at
org.elasticsearch.common.inject.assistedinject.FactoryProvider2.initialize(Unknown
Source)\n at _unknown_\n\n1 error]; ,
In my opnion, this error raises by ThaiWordFilterFactory which has`t a
zeo-argument constructor. In fact, the ThaiWordFilterFactory has only a
following constructor.
/** Creates a new ThaiWordFilterFactory */
public ThaiWordFilterFactory(MapString,String args) {
super(args);
assureMatchVersion();
if (!args.isEmpty()) {
throw new IllegalArgumentException(Unknown parameters: + args);
}
}
If you don`t mind, I have an one more question. Can I define a
constructor argument in above settings JSON.
2014년 2월 7일 금요일 오후 11시 17분 59초 UTC+9, Nikolas Everett 님의 말:
If you don't like the language analyzer you have to rebuild it as a
custom analyzer then add what you need to it.
{
analyzer: {
thai_with_ngram: {
type: custom,
tokenizer: standard,
filters: [standard, lowercase, thai, thai_stop, ngram]
}
},
filter: {
thai: {
type: org.apache.lucene.analysis.th.ThaiWordFilterFactory
},
thai_stop: {
type: stop,
stopwords_path: org/apache/lucene/analysis/th/stopwords.txt
},
ngram: { your ngram configuration here }
}
}
Builds it with your ngram configuration. I think. I'm taking quite a
few educated guesses here so I expect you to have to fiddle with it to get
it right.
How I did this:
1. Open the class called ThaiAnalyzer in the Lucene version
Elasticsearch is using and find the method called createComponents. For me
this is simple because I have Elasticsearch open in Eclipse.
2. That method defines the tokenizer (standard) and some filters
(standard, lowercase, ThaiWordFilter, and stop. You have to be able to
translate the class names to Elasticsearch's easier names to get this to
work properly.
3. Now build it as a custom filter with your extra filter in there.
That is thai_with_ngram above.
4. Next you'll need to define all the filters that don't exist by
default in Elasticsearch. In this case that is thai, thai_stop, and your
ngram filter. In order:
5. The thai filter doesn't have an easy Elasticsearch mapping so you
have to tell Elasticsearch the class name to load. That class doesn't take
an configuration so we're done.
6. The thai_stop filter is just a regular stop word filter with thai
stop words. But Elasticserach doesn't have an easy name to reference the
thai stop words file. That isn't too bad, as you can load the stopwords
file from the classepath. It lives in Lucene at the path I added above.
7. The ngram filter is yours to build but it is well documented.
That took longer then I expected but it was worth the exercise so I'll
remember how to do it again when I need it. For reference, I do it for
English which has more filters but they all have easy names.
Nik
On Fri, Feb 7, 2014 at 12:59 AM, Min Cha mins...@gmail.com wrote:
Hi folks.
I would like to develop for a searching system for Thai language.
First of all, I found Thai analyzer and it seemed like good.
Actually, but, It doesn`t meet my whole requirement.
I decided what extends it.
For example, I would like to add nGram token filter on the Thai
analyzer without any changes on it.
How to do this?
Please, give me some advice.
--
You received this message because you are subscribed to the Google
Groups elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/5041f397-8732-413f-8e50-46e25610c639%
40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To
Problem when migrate to 1.0
Hi, I yesterday moved my our cluster which have single node from 09.11 to 1.0 and I am getting following message from elasticsearch log [2014-02-14 11:14:40,528][WARN ][transport.netty ] [mipLoggingCenter.mi-pay.com] Message not fully read (request) for [0] and action [], resetting [2014-02-14 11:14:41,907][WARN ][transport.netty ] [mipLoggingCenter.mi-pay.com] Message not fully read (request) for [2] and action [], resetting [2014-02-14 11:14:43,429][WARN ][transport.netty ] [mipLoggingCenter.mi-pay.com] Message not fully read (request) for [4] and action [], resetting [2014-02-14 11:14:44,439][WARN ][transport.netty ] [mipLoggingCenter.mi-pay.com] Message not fully read (request) for [6] and action [], resetting On the logstash 1.3.3 side I am getting following messages log4j, [2014-02-14T11:14:40.576] WARN: org.elasticsearch.discovery.zen.ping.unicast: [Redeemer] failed to send ping to [[#zen_unicast_1#][inet[mipLoggingCenter.mi-pay.com/10.3.57.34:9300]]] org.elasticsearch.transport.RemoteTransportException: Failed to deserialize exception response from stream Caused by: org.elasticsearch.transport.TransportSerializationException: Failed to deserialize exception response from stream at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:169) at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:123) at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) Caused by: java.io.InvalidClassException: failed to read class descriptor at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1601) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1515) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1515) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1515) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1515) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1769) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:167) ... 23 more Caused by: java.lang.ClassNotFoundException: org.elasticsearch.ElasticsearchException at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at
Re: Problem when migrate to 1.0
Did you upgrade all nodes and restart them all? It sounds like node mipLoggingCenter.mi-pay.com/10.3.57.34 is still running a 0.90.x version. -- David Pilato | Technical Advocate | Elasticsearch.com @dadoonet | @elasticsearchfr Le 14 février 2014 à 12:22:11, Aamir Khan (aamir.maj...@gmail.com) a écrit: Hi, I yesterday moved my our cluster which have single node from 09.11 to 1.0 and I am getting following message from elasticsearch log [2014-02-14 11:14:40,528][WARN ][transport.netty ] [mipLoggingCenter.mi-pay.com] Message not fully read (request) for [0] and action [], resetting [2014-02-14 11:14:41,907][WARN ][transport.netty ] [mipLoggingCenter.mi-pay.com] Message not fully read (request) for [2] and action [], resetting [2014-02-14 11:14:43,429][WARN ][transport.netty ] [mipLoggingCenter.mi-pay.com] Message not fully read (request) for [4] and action [], resetting [2014-02-14 11:14:44,439][WARN ][transport.netty ] [mipLoggingCenter.mi-pay.com] Message not fully read (request) for [6] and action [], resetting On the logstash 1.3.3 side I am getting following messages log4j, [2014-02-14T11:14:40.576] WARN: org.elasticsearch.discovery.zen.ping.unicast: [Redeemer] failed to send ping to [[#zen_unicast_1#][inet[mipLoggingCenter.mi-pay.com/10.3.57.34:9300]]] org.elasticsearch.transport.RemoteTransportException: Failed to deserialize exception response from stream Caused by: org.elasticsearch.transport.TransportSerializationException: Failed to deserialize exception response from stream at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:169) at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:123) at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) Caused by: java.io.InvalidClassException: failed to read class descriptor at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1601) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1515) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1515) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1515) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1515) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1769) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at
Re: Marvel not collecting system metrics
May I ask which OS you are running ES on? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c840c19d-e04d-4f51-bd3f-4c8bc7d7fe0d%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Set the number_of_shards
There is no right (or wrong) answer. The best way is to test. Start with the default 5 shards and load real data into it at the rate that you expect in production. And then query it at the rate that you expect in production - check throughput and response times. Then run your facets, sorts, aggregations - check throughput, response times, and RAM usage. After a little but of testing, you should get a good sense of the limits of your hardware (per node) and then go from there. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3f908453-78e7-4af3-98fe-900a41ba0e91%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Elasticsearch global search different filter on multiple indexes..
I see, I didn't understand your original question. Unfortunately, you
cannot say something like highlight all fields except field A, B, C. You
can only list the fields you want highlighting on, and in addition you can
use wildcards when specifying field names like for example:
{
_source: false,
highlight: {
fields: {
ti*: {
number_of_fragments: 0
}
}
}
}
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/dc8a55b1-fbd4-431b-a808-c1bb6e2110a1%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
[ANN] Spring factories for elasticsearch 1.0
Heya, Just released elasticsearch spring factories project 1.0.0 for elasticsearch 1.0. I guess I can call this release the Valentine's day edition :-) https://github.com/dadoonet/spring-elasticsearch Older versions are in branch 0.x: https://github.com/dadoonet/spring-elasticsearch/tree/0.x Enjoy! -- David Pilato | Technical Advocate | Elasticsearch.com @dadoonet | @elasticsearchfr -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.52fe12cb.140e0f76.17b2%40MacBook-Air-de-David.local. For more options, visit https://groups.google.com/groups/opt_out.
Re: Kibana 3 index settings
I am looking for the same answer. ever got to find out how? On Thursday, January 2, 2014 3:50:59 AM UTC-5, spezam . wrote: Hello, in Kibana 3 is possible to set from the dashboard settings, the index settings. I'm using for this a day based timestamping, with an index pattern such as [dc1_logstash-].MM.DD Now, we added a new datacenter to the elasticsearch cluster, with different index names. Is it possible to use a regexp, or an array of indexes like in kibana2 Smart_index_pattern = ['dc1_logstash-%Y.%m.%d', 'dc2_logstash-%Y.%m.%d'] I tried with [*_logstash-].MM.DD] but I got a no indices error. Thanks, Matteo -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/698504ac-48ea-49ab-a092-9402815997e3%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Building custom panels in Kibana
It's actually not that difficult. Just need a little patience learning AngularJS. The easiest way to start is to look: 1) src/app/panels is where all the panels live - copy one out of here (I'd start with the text panel), create a new folder - new name based on your panel name, and edit and strip down the editor, module, and js files (just rename your panel name in the code accordingly) 2) src/config.js is where you will add your panel to make it visible to kibana. Scroll down to the bottom and add it to the list Assuming you got no syntax errors, clear your browser cache and refresh and you should be able to add your new panel onto the dashboard. If something is not working, just reverse all the processes above and you can go back to your original Kibana state without much problem. You'll probably want to do this in a DEV environment and test it first anyway. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1cd29006-23d1-4a30-b4d9-6ef53434a70b%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Search for a value across multiple fields
We're almost there!
This is the result of the query that I have posted:
- hits: {
- total: 3
- max_score: 4.724929
- hits: [
- {
- _index: website
- _type: structure
- _id: 7
- _score: 4.724929
- fields: {
- bologna_count: 0
}
}
- {
- _index: website
- _type: structure
- _id: 8
- _score: 4.724929
- fields: {
- bologna_count: 0
}
}
- {
- _index: website
- _type: structure
- _id: 6
- _score: 4.724929
- fields: {
- bologna_count: 0
}
}
]
}
in fact located all three records that contain bologna in the locality
and region, but bologna_count is always 0 and replicates the result 3
times.
For example:
the three records they found as locality:
bologna
bologna
Bolognano
What I would like is that I give back as a result: (similar to this)
hits: [
- {
- _index: website
- _type: structure
- _score: 4.724929
- fields: {
- count: 2
locality:bologna
}
}
- {
- _index: website
- _type: structure
- _score: 4.724929
- fields: {
- count: 1
locality:bolognano
}
}
]
so that it knows that there are 2 records with the name bologna and 1 as
bolognano.
Thanks
Il giorno venerdì 14 febbraio 2014 13:42:14 UTC+1, Binh Ly ha scritto:
I'm still not 100% sure I understand. Is this something that might work?
{
query: {
multi_match: {
query: bologna,
fields: [
locality,
region
]
}
},
script_fields: {
bologna_count: {
script: _index['locality']['bologna'].tf() + _index['region']['
bologna'].tf()
}
}
}
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/24d41fdb-2faf-4380-97c2-f0d56d4f0f20%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: Search for a value across multiple fields
Sorry, typo:
This is the result of the query *you posted*:
Il giorno venerdì 14 febbraio 2014 14:29:38 UTC+1, Luca Pau ha scritto:
We're almost there!
This is the result of the query that I have posted:
- hits: {
- total: 3
- max_score: 4.724929
- hits: [
- {
- _index: website
- _type: structure
- _id: 7
- _score: 4.724929
- fields: {
- bologna_count: 0
}
}
- {
- _index: website
- _type: structure
- _id: 8
- _score: 4.724929
- fields: {
- bologna_count: 0
}
}
- {
- _index: website
- _type: structure
- _id: 6
- _score: 4.724929
- fields: {
- bologna_count: 0
}
}
]
}
in fact located all three records that contain bologna in the locality
and region, but bologna_count is always 0 and replicates the result 3
times.
For example:
the three records they found as locality:
bologna
bologna
Bolognano
What I would like is that I give back as a result: (similar to this)
hits: [
- {
- _index: website
- _type: structure
- _score: 4.724929
- fields: {
- count: 2
locality:bologna
}
}
- {
- _index: website
- _type: structure
- _score: 4.724929
- fields: {
- count: 1
locality:bolognano
}
}
]
so that it knows that there are 2 records with the name bologna and 1 as
bolognano.
Thanks
Il giorno venerdì 14 febbraio 2014 13:42:14 UTC+1, Binh Ly ha scritto:
I'm still not 100% sure I understand. Is this something that might work?
{
query: {
multi_match: {
query: bologna,
fields: [
locality,
region
]
}
},
script_fields: {
bologna_count: {
script: _index['locality']['bologna'].tf() + _index['region']['
bologna'].tf()
}
}
}
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/e01434e0-be58-45a1-8373-9636219ca569%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: Kibana 3 index settings
Not yet, I'm still using Kibana 2 because of this issue. On Friday, February 14, 2014 2:21:38 PM UTC+1, Pascal Larivee wrote: I am looking for the same answer. ever got to find out how? On Thursday, January 2, 2014 3:50:59 AM UTC-5, spezam . wrote: Hello, in Kibana 3 is possible to set from the dashboard settings, the index settings. I'm using for this a day based timestamping, with an index pattern such as [dc1_logstash-].MM.DD Now, we added a new datacenter to the elasticsearch cluster, with different index names. Is it possible to use a regexp, or an array of indexes like in kibana2 Smart_index_pattern = ['dc1_logstash-%Y.%m.%d', 'dc2_logstash-%Y.%m.%d'] I tried with [*_logstash-].MM.DD] but I got a no indices error. Thanks, Matteo -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6e143a28-db8a-4b2a-9c84-8f1d2cb80744%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
nested query doesn't work
I have the following setup:
indexes:
website:
client: default
finder: ~
settings:
index:
analysis:
analyzer:
my_analyzer:
type: custom
tokenizer: lowercase
filter : [my_ngram]
filter:
my_ngram:
type: edgeNGram
min_gram: 2
max_gram: 5
types:
shop:
mappings:
username: { boost: 3 }
fullname: { boost: 2 }
bio: { boost: 1 }
userPictures:
type: nested
properties:
caption: { boost: 5 }
persistence:
driver: orm
model: App\MainBundle\Entity\InstagramShop
provider: ~
listener: ~
and here's my nested query in PHP:
$finder =
$this-container-get('fos_elastica.finder.website');
$mainQuery = new \Elastica\Query\Bool();
$shopQuery = new \Elastica\Query\QueryString();
$shopQuery-setParam('query', $searchquery);
$shopQuery-setParam('fields', array(
'username', 'bio', 'fullname'
));
$pictureQuery = new \Elastica\Query\QueryString();
$pictureQuery-setParam('query', $searchquery);
$pictureQuery-setParam('fields', array(
'caption'
));
$nestedQuery = new \Elastica\Query\Nested();
$nestedQuery-setPath('userPictures');
$nestedQuery-setQuery($pictureQuery);
$mainQuery-addShould($shopQuery);
$mainQuery-addShould($nestedQuery);
$items = $finder-find($mainQuery);
I wonder why this only matches with the shopQuery and not with the
nestedQuery. Here's the structure for the InstagramShop class:
class InstagramShop
{
/**
* @Exclude()
* @ORM\OneToMany(targetEntity=InstagramShopPicture, mappedBy=shop,
cascade={persist})
* @ORM\OrderBy({created = DESC})
*/
protected $userPictures;
/**
* Get attributes
*
* @return \Doctrine\Common\Collections\Collection
*/
public function getUserpictures()
{
return $this-userPictures;
}
}
User pictures here is a collection of InstagramShopPicture which has a
caption of type string
Any idea why it can't find userPictures caption?
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/9277d215-7feb-4087-a08b-ef7b8a4d9ca9%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: How to set the edit distance of the fuzzy like this query?
It should be the fuzziness property: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-flt-query.html More details about how you can customize it here: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/common-options.html#fuzziness -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1ba19644-129e-432b-b4b7-0d33a3c0c48f%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Getting started on a Mac (Splunk user looking to migrate)
Hello, I've updated the info on my page which contains fairly old info (approx. ES 0.90.3) *https://en.opensuse.org/User:Tsu2/Install_and_Intro_Logstash-Elasticsearch-Kibana*https://en.opensuse.org/User:Tsu2/Install_and_Intro_Logstash-Elasticsearch-Kibana to http://en.opensuse.org/User:Tsu2/elasticsearch_1.0#Running_Logstash In the new, updated page I preview the content in the Logstash tutorials. Some stuff I cover The different options how to install and run Logstash, Elasticsearch, Kibana for the tutorials Breaks out a number of items which are often shown in the tutorials but with no description or highlighting. Stuff that's easy to overlook or confuse. As Mark describes, Since no install pattern has been created for running on OSX, it's still fairly simple to invoke all three apps by running only Logstash or downloading and running the jar files. Jar files can be run from any location simply by command (no installation). You only need a suitable Java JRE (Oracle or openJDK 1.7 ordinarily should be sufficient). Tony On Thursday, February 13, 2014 6:59:27 PM UTC-8, Mark Walkom wrote: OS X isn't a fully supported OS, in that there are no pkg's for the 3 components, so you'll have to run the jar/bin's after grabbing the tgz packages from the site if you want to do it in your native environment. I'd recommend that you spin up a linux VM and try it in there as it's a simpler process. Here's a few run throughs that a search turned up; https://en.opensuse.org/User:Tsu2/Install_and_Intro_Logstash-Elasticsearch-Kibana http://www.jaddog.org/2014/01/16/openstack-logstash-elasticsearch-kibana/ http://docs.fluentd.org/articles/free-alternative-to-splunk-by-fluentd http://www.vmdoh.com/blog/centralizing-logs-lumberjack-logstash-and-elasticsearch Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com javascript: web: www.campaignmonitor.com On 14 February 2014 13:37, Phoenix Kiula phoeni...@gmail.comjavascript: wrote: Thank you for this. But that page, including the getting started guide, is nowhere close to what I requested. I need to install ElasticSearch, then LogStash and Kibana -- which I understand is the set of tools I need to get anywhere close to Splunk or SumoLogic. Right? Where's the guide to install all of them on Mac OSX and start playing around with localhost, and coding in PHP? Apologies if this sounds onerous, but my request is fairly simple and straightforward even for an open source tool. There's a lot of effort on the ES website to showcase case studies and whatnot. A bit of effort in helping people actually get started without a PhD would be great too! Maybe I'm just missing something. Thanks On Friday, February 14, 2014 10:02:24 AM UTC+8, Mark Walkom wrote: ES is free as it's open source. You might want to take a look here http://logstash.net/docs/1.3.3/learnas it has a good getting started guide. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 14 February 2014 12:52, Phoenix Kiula phoeni...@gmail.com wrote: Hi. I'm somewhat familiar with Splunk, because our company invested in it. Now it's proving to be expensive so we're looking for alternatives. (It's a super tool though) ElasticSearch was recommended on Quora. But downloading and trying to install it is hideous. This seems for tech nerds. Or am I missing something? I'm on a Mac OSX and want to try it out. The Splunk or SumoLogic installs were 10 seconds and I was up and running. Could someone please point me to a simple guide to: 1. Install the whole thing: including LogStash and Kibana, because just searching is not what I wish to do, I want to make sense of the data and therefore dashboarding is important. I couldn't find a simple step by step tutorial or guide on the website to install this whole stack. Google turns up stuff like this -- http://red-badger.com/blog/ 2013/11/08/getting-started-with-elasticsearch/ -- which is ridiculously dysfunctional. (And I don't want to install brew just to install simple stuff.) 2. Secondly, once installed, I'd like to know how to simply get the data into the index from: a) a MySQL database b) a folder with XML files...I want to slurp all the files into the index, and then incrementally only get new files in the future c) a folder with text log files...and same incremental auto-pickup as (b) above Don't seem to find a simple non-geeky guide to do this. I'm a programmer in PHP and web technologies (JS etc), and manage my own dedicated Linux hosting, so not averse to code or commands, but want to find some coherent and simple guide. 3. Thirdly, the demo.kibana.org is hideous. It looks like the best way to UN-sell this technology. Is there a better place I could see actual Kibana in use, you know, with a proper
Re: 0.90.11 stuck with high memory usage during bulk indexing and even hours after stopping
Thanks both for your input.
@Jörg:
I understand ES uses all available process memory. I meant jvm memory
usage, which it tries to reclaims when it exceeds 75% (due
to -XX:CMSInitiatingOccupancyFraction=75) option.
I don't know what kind of queries use Lucene FST, could you be kind enough
to explain. I also didn't know about bloom filter and it's
memory usage, is their a way to check how much memory usage it's adding.
I will update JVM, but the issue is the same bulk indexing was not making
node out of memory in v0.90.7, it's doing it with v0.90.11
@*Adrien *:
I will play with merge throttling to speed it up. After many hours, even
after merge operations are finished, the memory still wasn't
reclaimed so I am more worried about that.
fyi, from ES logs -
[2014-02-14 10:09:54,109][WARN ][monitor.jvm ] [machine1.node2]
[gc][old][75611][2970] duration [43s], collections [1]/[44.1s], total [43s
]/[55.5m], memory [11.3gb]-[10.6gb]/[11.8gb], all_pools {[young] [454.6mb
]-[10.4mb]/[865.3mb]}{[survivor] [108.1mb]-[0b]/[108.1mb]}{[old] [10.8gb
]-[10.6gb]/[10.9gb]}
And from /_cluster/stats request -
fielddata : {
memory_size : 3.6gb,
memory_size_in_bytes : 3881191105,
evictions : 0
},
filter_cache : {
memory_size : 622.4mb,
memory_size_in_bytes : 652677071,
evictions : 0
},
id_cache : {
memory_size : 2gb,
memory_size_in_bytes : 2170019078
},
completion : {
size : 0b,
size_in_bytes : 0
},
segments : {
count : 789,
memory : 3.4gb,
memory_in_bytes : 3730255779
}
If node is running out of memory, shouldn't ES be reclaiming id_cache or
fielddata ?
On Thursday, February 13, 2014 10:19:28 AM UTC-5, Ankush Jhalani wrote:
We have a single node, 12GB, 16 core ES instance to which we are 12
threads bulk indexing into a 12shard index. Each thread sends a request of
size kb to couple megabytes. The thread bulk queue_size is increased from
default 50 to 100.
With v0.90.11, we are noticing that the jvm memory usage keeps growing
slowly and doesn't go down, gc runs frequently but doesn't free up much
memory. From debug logs, it seems the segment merges are happening. However
even after we stop indexing, for many hours the instance is busy doing
segment merges. Sample gist from hot threads I ran couple minutes apart - (
https://gist.github.com/ajhalani/8976792). Even after 16 hours and little
use on the machine, the jvm memory usage is about 80% (CMS should run at
75%) and nodes stats show is running very frequently.
If we don't stop indexing, eventually after 60-70GB indexing the instance
goes out of memory. This seems like a memory leak, we didn't face this
issue with 0.90.7 (though we were probably using a 6 thread process for
bulk indexing).
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/99b4d682-5d0d-4255-bf5f-ce0561b111be%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: Kibana 3 index settings
Can you try something like this: [logstash-].MM.DD,[dc1_logstash-].MM.DD,[dc2_logstash-].MM.DD -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c43a1533-edbe-4d65-9be9-dba173f6f076%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: 0.90.11 stuck with high memory usage during bulk indexing and even hours after stopping
I ran '_cache/clear' which cleaned up fielddata, id_cache and jvm memory usage dropped ~10.5GB - ~5 GB.. Shouldn't ES itself clear up these cache when jvm memory usage becomes really high? I see the gc count kept increasing but not a lot of memory was reclaimed until I ran _cache/clear.. On Thursday, February 13, 2014 10:19:28 AM UTC-5, Ankush Jhalani wrote: We have a single node, 12GB, 16 core ES instance to which we are 12 threads bulk indexing into a 12shard index. Each thread sends a request of size kb to couple megabytes. The thread bulk queue_size is increased from default 50 to 100. With v0.90.11, we are noticing that the jvm memory usage keeps growing slowly and doesn't go down, gc runs frequently but doesn't free up much memory. From debug logs, it seems the segment merges are happening. However even after we stop indexing, for many hours the instance is busy doing segment merges. Sample gist from hot threads I ran couple minutes apart - ( https://gist.github.com/ajhalani/8976792). Even after 16 hours and little use on the machine, the jvm memory usage is about 80% (CMS should run at 75%) and nodes stats show is running very frequently. If we don't stop indexing, eventually after 60-70GB indexing the instance goes out of memory. This seems like a memory leak, we didn't face this issue with 0.90.7 (though we were probably using a 6 thread process for bulk indexing). -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f53af0c2-3d30-4059-a044-54213f1a32f3%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Optimize
Hello, I want to know when and if I should manually call optimize on elasticsearch. This blog seems to say it's a bad idea: http://gibrown.wordpress.com/2013/01/24/elasticsearch-five-things-i-was-doing-wrong/ However, there must be a reason for optimize to be exposed in the rest api. []'s Rafael -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKCnWVm9GTpo%2B79uCCfgfr9Tej98M5UyiB02BDTJ-8EJyMv%3Djg%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Kibana Filtered Queries Failing After ES 1.0 Upgrade
Yes - I downloaded the master from GitHub and was running still seeing the issue. On Friday, February 14, 2014 10:38:49 AM UTC-5, Binh Ly wrote: I'm curious if you're running the latest Kibana, or an older one. The field query has been deprecated (and removed in ES 1.0) which is the cause of your error: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/_deprecations.html -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a4a88f67-e5b8-4945-ad56-586ac5b34b48%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Marvel houskeeping
I upgraded elasticsearch to 0.90.11 and installed marvel. Congratulations on a really nice tool! Now I have a small issue: since marvel is generating quite a lot of data (for our develop system), I would like to configure an automatic delete of old data. Is there such an option? I didn't find anything in the documentation. It would be great to specify a rolling window of n days of data to keep. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/90ac3f1f-23c4-461f-95d5-f054f1fc5706%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Kibana Filtered Queries Failing After ES 1.0 Upgrade
Note that I'm using the past tense only because I reverted back to ES 0.90.9, not because I figured out how to solve the issue :) -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7c9221a1-0866-4e46-a345-92d99cc176d0%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Marvel not collecting system metrics
On Friday, February 14, 2014 4:06:06 AM UTC-8, Binh Ly wrote: May I ask which OS you are running ES on? IIRC Docker is a management tool for LXC. So, it does pique the thoughts... Where should marvel be pulling stats in an LXC deployment? It's not fully isolated like other virtualization technologies and access to various Host processes are blocked. I suppose it might be possible to point to statistics derived by Docker, but that probably requires special configuration. Some thoughts, Tony -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b409c6f0-1ea3-44f1-bffc-423edc4bb792%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Automatic Range Filter. Possible ?
Would the histogram facet work for you? http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-facets-histogram-facet.html BTW, if you've upgraded to 1.0, you might want to look at aggregations which are more powerful than facets. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/308d0542-b43c-49b9-8d08-ebd4eafe1ffe%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Kibana Filtered Queries Failing After ES 1.0 Upgrade
Strange. Just want to confirm your behavior again. I just downloaded Kibana 3 and I hover on the top left and says Kibana 3 milestone pre 5. I clear all my browser cache to ensure there is no old Kibana code lurking around. Then I create a new logstash dashboard. Then I go to the filtering section and add a new filter. I get a new filter and it says querystring, must, and a query box (there are no other options/choices). I do not see the ability to add a field filter though. Can you confirm if we have the same behavior or something you're seeing is different from mine? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9bd93b6e-a6a7-4dab-8d66-c88c25feea6f%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Automatic Range Filter. Possible ?
There is no automatic bucketing in Elasticsearch. I mimic the behavior with an expensive process that uses many smaller fixed ranges which are reduced into the number of buckets needed on the client side. Easily the slowest part of my query. My goal was to wait for the facet refactor (which has since become aggregations) before attempting a better server side implementation, but now I am too busy to revisit it. Cheers, Ivan On Fri, Feb 14, 2014 at 7:45 AM, Georges@Bibtol s...@quable.fr wrote: Hi, This could work but the issue is the same. I do not want to tell the interval but I want elastic to choose for me. Le vendredi 14 février 2014 16:42:48 UTC+1, Binh Ly a écrit : Would the histogram facet work for you? http://www.elasticsearch.org/guide/en/elasticsearch/ reference/current/search-facets-histogram-facet.html BTW, if you've upgraded to 1.0, you might want to look at aggregations which are more powerful than facets. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d15ad9a0-edb0-4fea-bf35-0d8674b4a924%40googlegroups.com . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDT%3DRoR6U0pr8c365m6Ef%3D-%3D5eOYKeoszg%2B6WXXCu6H6w%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
ElasticSearch 1.0 and which version of LogStach?
I am a new user of Elasticsearch and Logstach. I have downloaded new versions of these tools (directly from ElasticSearch download page, exacly: Elasticsearch 1.0.0 and Logstash 1.3.3). After running both I noticed that Elasticsearch throws such exceptions like below. Is it possible to run these tools together? How? Which version of Logstach I have to use with ElasticSearch 1.0? I can't use lower version of ElasticSearch - I need built-in snapshot/restore module. *[2014-02-14T17:17:49.821] WARN: org.elasticsearch.discovery.zen.ping.unicast: [Rasputin, Illyana] failed to send ping to [[#zen_unicast_1#][inet[/127.0.0.1:9300]]]org.elasticsearch.transport.RemoteTransportException: Failed to deserialize exception response from streamCaused by: org.elasticsearch.transport.TransportSerializationException: Failed to deserialize exception response from streamat org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:169) at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:123) at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)at java.lang.Thread.run(Unknown Source)Caused by: java.io.InvalidClassException: failed to read class descriptorat java.io.ObjectInputStream.readNonProxyDesc(Unknown Source)at java.io.ObjectInputStream.readClassDesc(Unknown Source)at java.io.ObjectInputStream.readNonProxyDesc(Unknown Source)at java.io.ObjectInputStream.readClassDesc(Unknown Source)at java.io.ObjectInputStream.readNonProxyDesc(Unknown Source)at java.io.ObjectInputStream.readClassDesc(Unknown Source)at java.io.ObjectInputStream.readNonProxyDesc(Unknown Source)at java.io.ObjectInputStream.readClassDesc(Unknown Source)at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)at java.io.ObjectInputStream.readObject0(Unknown Source)at java.io.ObjectInputStream.readObject(Unknown Source)at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:167) ... 23 moreCaused by: java.lang.ClassNotFoundException: org.elasticsearch.ElasticsearchExceptionat java.net.URLClassLoader$1.run(Unknown Source)at java.net.URLClassLoader$1.run(Unknown Source)at java.security.AccessController.doPrivileged(Native Method)at java.net.URLClassLoader.findClass(Unknown Source)at org.jruby.util.JRubyClassLoader.findClass(JRubyClassLoader.java:128)
Re: NumberFormatException when sorting by numeric document ID
This can *absolutely* be fixed in ElasticSearch. It's not a problem with Lucene, but with how ES data is mapped onto the Lucene data model. The problem is that types and fields use local names instead of fully-qualified names. As far as Lucene is concerned, there would be a field named user.id mapped as a long, another field named product.id mapped as a string, and a nested type named user.address.id mapped as an integer. Under this kind of system, user and product can exist in the same index, without even the possibility that their names and types would clash. benji On Thursday, February 13, 2014 6:41:41 PM UTC-5, Ivan Brusic wrote: I doubt this issue will ever be fixed since the limitation exists in Lucene. All types belong to the same index and a field's data needs to be uniform in Lucene's eyes. A document's type is used to indicate different mappings for a document, but not different ways to segment the data types in the index itself. This scenario should be documented however, so that others do not fall into the same trap. -- Ivan On Thu, Feb 13, 2014 at 9:18 AM, Benji Smith be...@benjismith.netjavascript: wrote: Thanks for your comment! Looks like correct github issue to reference is this one: https://github.com/elasticsearch/elasticsearch/issues/4081 I've added my comments, and I'm rooting for a solution to this problem rather than just a warning, which won't really solve the problem for us. Fingers crossed! benji On Thursday, February 13, 2014 11:21:08 AM UTC-5, Binh Ly wrote: This is on the plate. I'm not 100% sure exactly what the fix will be but it could be something along the lines of a warning when a mapping is introduced with the same field name but different types. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com javascript:. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c1cb37bc-1698-4b03-9791-535e9f12bbf0%40googlegroups.com . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2934c18e-fce5-4ffa-9fe4-b0115d53e2f9%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Kibana 3 index settings
Strange, I just downloaded the latest Kibana, and I created 2 simple
logstash indexes, logstash-2014.01.29 and a_logstash-2014.01.29. Then I
went into Kibana with a new dashboard and set the index timestampping to
day and pattern to [a_logstash-].MM.DD,[logstash-].MM.DD
My histogram shows some data and when I inspect the query, it says:
curl -XGET
'http://localhost:9200/logstash-2014.01.29,a_logstash-2014.01.29/_search?pretty'
-d '{...}'
I'm curious, which version of Kibana are you using?
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ef7a4b55-9cc7-45d8-b783-7df6bf406253%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: ElasticSearch 1.0 and which version of LogStach?
For now, you can use the elasticsearch_http output (instead of
elasticsearch) and you should be able to get LS 1.3.3 going with ES 1.0.
For example:
output {
elasticsearch_http {
host = localhost
}
}
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d0de1889-e1f8-443f-bc5f-c80662d78bf9%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Azure Cloud Plugin incompatible with Elasticsearch 1.0.0 GA
Hi,
I have a small elasticsearch 0.9.11 cluster in Windows Azure that uses the
Azure Cloud Plugin for node discovery.
When trying to upgrade to elasticsearch 1.0.0 today, I noticed that when I
active the Azure Cloud Plugin, elasticsearch won't start :
[2014-02-14 16:47:22,987][ERROR][bootstrap] {1.0.0}:
Initialization Failed ...
- ExecutionError[org.elasticsearch.common.util.concurrent.ExecutionError:
java.lang.NoClassDefFoundError: org/elasticsearch/ElasticSearchException]
ExecutionError[java.lang.NoClassDefFoundError:
org/elasticsearch/ElasticSearchException]
NoClassDefFoundError[org/elasticsearch/ElasticSearchException]
ClassNotFoundException[org.elasticsearch.ElasticSearchException]
If I remove the azure cloud plugin, everything works fine.
Is this a known issue and is there a new version of the Azure Cloud Plugin
on the way that will fix this issue?
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/213c2fce-b971-4313-b778-ebaba5c9870d%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: Azure Cloud Plugin incompatible with Elasticsearch 1.0.0 GA
Ah! You're right. I need to release it soon.
By now, you can try 2.0.0.RC1-SNAPSHOT as the version name. It should work.
--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr
Le 14 février 2014 à 18:19:00, Hugo Leclerc (hugo.lecl...@radio-canada.ca) a
écrit:
Hi,
I have a small elasticsearch 0.9.11 cluster in Windows Azure that uses the
Azure Cloud Plugin for node discovery.
When trying to upgrade to elasticsearch 1.0.0 today, I noticed that when I
active the Azure Cloud Plugin, elasticsearch won't start :
[2014-02-14 16:47:22,987][ERROR][bootstrap ] {1.0.0}:
Initialization Failed ...
- ExecutionError[org.elasticsearch.common.util.concurrent.ExecutionError:
java.lang.NoClassDefFoundError: org/elasticsearch/ElasticSearchException]
ExecutionError[java.lang.NoClassDefFoundError:
org/elasticsearch/ElasticSearchException]
NoClassDefFoundError[org/elasticsearch/ElasticSearchException]
ClassNotFoundException[org.elasticsearch.ElasticSearchException]
If I remove the azure cloud plugin, everything works fine.
Is this a known issue and is there a new version of the Azure Cloud Plugin on
the way that will fix this issue?
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/213c2fce-b971-4313-b778-ebaba5c9870d%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/etPan.52fe5090.79e2a9e3.f2%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/groups/opt_out.
Re: 1.0.0 maven repo
Here? https://oss.sonatype.org/index.html#nexus-search;gav~org.elasticsearch~elasticsearch~1.0.0~~ -- David Pilato | Technical Advocate | Elasticsearch.com @dadoonet | @elasticsearchfr Le 14 février 2014 à 18:08:07, Robin Verlangen (ro...@us2.nl) a écrit: Hi there, Where can I find the 1.0.0 release in maven? It seems that the latest one is still the RC http://mvnrepository.com/artifact/org.elasticsearch/elasticsearch Thank you in advance! Best regards, Robin Verlangen Chief Data Architect W http://www.robinverlangen.nl E ro...@us2.nl What is CloudPelican? Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CADVHTB9gVxueOVSyFqj5odcmXCU-MNWH6iosRk9KYAOdbyHAzg%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.52fe5107.12200854.f2%40MacBook-Air-de-David.local. For more options, visit https://groups.google.com/groups/opt_out.
Re: Azure Cloud Plugin incompatible with Elasticsearch 1.0.0 GA
Works like a charm. Thanks!
On Friday, February 14, 2014 12:21:19 PM UTC-5, David Pilato wrote:
Ah! You're right. I need to release it soon.
By now, you can try 2.0.0.RC1-SNAPSHOT as the version name. It should work.
--
*David Pilato* | *Technical Advocate* | *Elasticsearch.com*
@dadoonet https://twitter.com/dadoonet |
@elasticsearchfrhttps://twitter.com/elasticsearchfr
Le 14 février 2014 à 18:19:00, Hugo Leclerc
(hugo.l...@radio-canada.cajavascript:)
a écrit:
Hi,
I have a small elasticsearch 0.9.11 cluster in Windows Azure that uses the
Azure Cloud Plugin for node discovery.
When trying to upgrade to elasticsearch 1.0.0 today, I noticed that when I
active the Azure Cloud Plugin, elasticsearch won't start :
[2014-02-14 16:47:22,987][ERROR][bootstrap] {1.0.0}:
Initialization Failed ...
- ExecutionError[org.elasticsearch.common.util.concurrent.ExecutionError:
java.lang.NoClassDefFoundError: org/elasticsearch/ElasticSearchException]
ExecutionError[java.lang.NoClassDefFoundError:
org/elasticsearch/ElasticSearchException]
NoClassDefFoundError[org/elasticsearch/ElasticSearchException]
ClassNotFoundException[org.elasticsearch.ElasticSearchException]
If I remove the azure cloud plugin, everything works fine.
Is this a known issue and is there a new version of the Azure Cloud Plugin
on the way that will fix this issue?
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com javascript:.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/213c2fce-b971-4313-b778-ebaba5c9870d%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/96e151f6-2ad9-4eb8-8293-1829d1a5e7ca%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
[ANN] elasticsearch-cloud-azure 2.0.0 for elasticsearch 1.0.0
Heya, Just released elasticsearch-cloud-azure 2.0.0 for elasticsearch 1.0.0. https://github.com/elasticsearch/elasticsearch-cloud-azure Next version (2.1.0) will add Azure Cloud Storage repository for Snapshot and Restore. Contributions/PR/Issues/Doc warmly welcomed! :-) -- David Pilato | Technical Advocate | Elasticsearch.com @dadoonet | @elasticsearchfr -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.52fe56f9.1190cde7.f2%40MacBook-Air-de-David.local. For more options, visit https://groups.google.com/groups/opt_out.
Re: Kibana 3 index settings
My previous installation was around 3 weeks old, after upgrading to the
latest Kibana it seems it started working just great.
Thanks a million!
On Friday, February 14, 2014 5:55:03 PM UTC+1, Binh Ly wrote:
Strange, I just downloaded the latest Kibana, and I created 2 simple
logstash indexes, logstash-2014.01.29 and a_logstash-2014.01.29. Then I
went into Kibana with a new dashboard and set the index timestampping to
day and pattern to [a_logstash-].MM.DD,[logstash-].MM.DD
My histogram shows some data and when I inspect the query, it says:
curl -XGET
'http://localhost:9200/logstash-2014.01.29,a_logstash-2014.01.29/_search?pretty'
-d '{...}'
I'm curious, which version of Kibana are you using?
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3bd3a91d-6c03-480d-9bcf-f57972fa8651%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: Shards allocation in cluster on the same node
Shards should distribute over the 2 nodes assuming they are part of a single cluster. Theoretically, yes more shards *distributed across multiple nodes* will increase indexing speed. But you can still be limited by other resources such as network, CPU, memory so it's hard to say how much exactly will your throughput be. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/292bc034-f884-4373-be5d-5cb87b1fded4%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Kibana Filtered Queries Failing After ES 1.0 Upgrade
Binh, First let me thank you for helping me track down what’s going on here. So I can confirm that everything I see is the same as what you saw (with the exception that at the end of the Kibana version mine says [master]). If I enter a query for: querystring must query : action:connect …then the expected results come back. Try this now: Go to your logstash dashboard and click one of the rows. Click the magnifying glass near one of the fields and see if you get back results. The above is how my users are the primary way that my users add filters to find what they’re looking for, and that’s the part that isn’t working for me. Thanks, Chris -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.52fe5cf3.79e2a9e3.e8%40april.sos.its.psu.edu. For more options, visit https://groups.google.com/groups/opt_out.
Re: Marvel houskeeping
I haven't thought of automating, but it seems to me that it should be easy to address manually. Haven't looked to see I this can be done in Marvel, but in elasticsearch-head and elasticsearch-hq both display the indices. Since data is indexed by data, you can select the indexes you wish and perform an operation on them. Of course the underlying curl command should be an option, too. You may be able to do the same in Marvel. Indices are displayed but I haven't investigated Marvel's ability to perform operations, I've only been looking at what can be configured to display. Tony On Friday, February 14, 2014 7:41:36 AM UTC-8, Thomas Andres wrote: I upgraded elasticsearch to 0.90.11 and installed marvel. Congratulations on a really nice tool! Now I have a small issue: since marvel is generating quite a lot of data (for our develop system), I would like to configure an automatic delete of old data. Is there such an option? I didn't find anything in the documentation. It would be great to specify a rolling window of n days of data to keep. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0bb77086-abc7-4234-a247-d1a5ce50ae8f%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Kibana Filtered Queries Failing After ES 1.0 Upgrade
Chris,
I tried your suggestion, in the table panel, I opened 1 row, and then
filtered (magnifying glass) on 1 field. It indeed added a field filter -
must, field, and value. However, it re-executed all the queries properly.
The new filter translated to this part in the query (which looks valid to
me):
{
fquery: {
query: {
query_string: {
query: _type:(\apache\)
}
},
_cache: true
}
}
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/56c7f1d8-8936-4622-baf9-068b035b424c%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: Shards allocation in cluster on the same node
I managed to split the shards by restarting ES on the master, then retested. Throughput is the same. 4500/sec seems a bit low, each doc is just 8k. Network doesn't seems to be the bottleneck. I check the IO on disk, and it's between 0 (probably when it's buffering before flushing, and 50/70). Do you think I should get Provisionned IO on my EC2 instance ? On Friday, February 14, 2014 12:53:11 PM UTC-5, Binh Ly wrote: Shards should distribute over the 2 nodes assuming they are part of a single cluster. Theoretically, yes more shards *distributed across multiple nodes* will increase indexing speed. But you can still be limited by other resources such as network, CPU, memory so it's hard to say how much exactly will your throughput be. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fd2c61ca-8e6a-4961-8dfd-ea5c7cb4b563%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Marvel not collecting system metrics
Hi Boaz,
fs: {
data: [
{
path: /data/elasticsearch/shared/docker/nodes/0
}
],
Does not appear to be. This field is populated when I run it without
Docker; so is it expecting any particular file to exist like /etc/fstab or
/etc/mtab?
Thanks!
On Fri, Feb 14, 2014 at 7:35 AM, Boaz Leskes b.les...@gmail.com wrote:
Hi scott
next to Binh's question can you check you have the following field in your
nodes stats( GET _node/stats): fs.total.available_in_bytes ?
On Friday, February 14, 2014 1:06:06 PM UTC+1, Binh Ly wrote:
May I ask which OS you are running ES on?
--
You received this message because you are subscribed to a topic in the
Google Groups elasticsearch group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/dhpxaOuoZWI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/466b1bd5-173d-46df-a8e2-39d52f2b8146%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CALDJu4U%3D__oqEDGVQQcUGK%2Ba9vvKMu8deJEEtrRRyDO7zW0BCw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: Shards allocation in cluster on the same node
I provisioned an IO 300 disk, no improvement at all. Logstash is running on the same instance as the master node. On Friday, February 14, 2014 2:10:47 PM UTC-5, Bastien Chong wrote: I managed to split the shards by restarting ES on the master, then retested. Throughput is the same. 4500/sec seems a bit low, each doc is just 8k. Network doesn't seems to be the bottleneck. I check the IO on disk, and it's between 0 (probably when it's buffering before flushing, and 50/70). Do you think I should get Provisionned IO on my EC2 instance ? On Friday, February 14, 2014 12:53:11 PM UTC-5, Binh Ly wrote: Shards should distribute over the 2 nodes assuming they are part of a single cluster. Theoretically, yes more shards *distributed across multiple nodes* will increase indexing speed. But you can still be limited by other resources such as network, CPU, memory so it's hard to say how much exactly will your throughput be. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a56f8f9c-8b09-4980-9b84-210522fe7300%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Setting GC for ES
I'm a little clueless when it comes to java options in ES, and was wondering where I define things like GC? I did try setting it in ES_JAVA_OPTS under /etc/default/elasticsearch, however when I did ES wouldn't start, so either my syntax is wrong, or something else. Can anyone provide a few pointers for me? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624b7g4YwRDfeHU1OJG9FBHmGO6Zt1UK%3DTd67Wx9K%2BhKq5g%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Index template.
What is the importance of Index template? Could someone explain with an example? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b2d425bf-8204-4fa3-ad02-2a9202fb2d16%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Marvel houskeeping
Hi Thomas, Marvel itself doesn't have a setting for this, but you can have a look at this tool, built by the logstash team to help management indices with time based data: https://github.com/elasticsearch/curator Cheers, Boaz On Friday, February 14, 2014 4:41:36 PM UTC+1, Thomas Andres wrote: I upgraded elasticsearch to 0.90.11 and installed marvel. Congratulations on a really nice tool! Now I have a small issue: since marvel is generating quite a lot of data (for our develop system), I would like to configure an automatic delete of old data. Is there such an option? I didn't find anything in the documentation. It would be great to specify a rolling window of n days of data to keep. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/98be7697-3788-42b3-9477-b55971bb3d2f%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Shards allocation in cluster on the same node
It's hard to diagnose things offline, but is it possible for you to run another logstash somewhere else (like maybe on the second box) and both of them in parallel and see what your combined ES throughput is. So they would be both writing to the same single ES cluster. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/aa945a7d-532c-4348-bc87-5d37bc7f1cd9%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Documents per shard
I've got indexes storing the same kind of data split into weekly chunks - there has been some fairly substantial variation in data volume. I've got a mapping change I need to make across all the back data, and I'm thinking it might make sense to try to rebalance the documents per shard so that I have around 1 shard per N documents. Is that a worthwhile time investment in terms of query performance, or should I just stick with the 3 shards per index I've been using so far? I'd keep 3 shards as a minimum, so if there's a week with 10 documents it would still have 3 shards. If I have an index that would end up with more than one shard per data node, does it make more sense to limit the number of shards to the number of data nodes, or go ahead and follow the 1 shard per N documents pattern? Thanks! Josh -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d775acd2-9717-4fe6-bbda-9c6d42f0cb39%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Automatic Range Filter. Possible ?
Here is a previous discussion on Rice/Sturges: https://groups.google.com/forum/#!msg/elasticsearch/CAZhIHtB1UI/Exzd2_DanbAJ Never did sit down and finally understand the paper Jörg linked. :) I really should find the time to revisit the issue since my implementation is costly. Ivan On Fri, Feb 14, 2014 at 11:46 AM, Brian Hudson brian.r.hud...@gmail.comwrote: When creating histograms for my UI if the user does not provide the number of bins/interval I'm finding the min and max values using the new min max aggregations and then computing the number of bins/interval using the Rice Rule described here: http://en.wikipedia.org/wiki/Histogram#Number_of_bins_and_width I'm pretty happy with the results. Brian On Friday, February 14, 2014 9:31:58 AM UTC-5, Georges@Bibtol wrote: Hi everyone, I have multiple facets on text, integer, date. I use range filter on some integer facets but I have to manually set the range (from x to y, from y to z, ...) Is there a way to ask ES to do those steps automatically ? I undersand that statical can help whit the mean value ... but it means to do 2 call just for one facets. Did I miss something ? Thanks for any clue and have everyone a good day. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/593570b9-3b31-473f-8e71-c1af49bbae1b%40googlegroups.com . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQA-WKoAdf0%3D8Pq9RAqX_%3D%3DVs-ekh5Espf66T-Wez7AZKw%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Marvel not collecting system metrics
It does indeed sound like some metrics are not available from your environment. Marvel/ES uses Sigar to collect these metrics (https://support.hyperic.com/display/SIGAR/Home). Each OS has different ways to provide (or not provide) these metrics. If you absolutely cannot get these metrics, you can create a Marvel index template (http://www.elasticsearch.org/guide/en/marvel/current/) and introduce these missing fields that Sigar cannot collect from your environment. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/886a1c50-4eb6-4a87-99a2-d2916032e04d%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Index template.
If you find yourself repeatedly creating indexes with some similar
characteristics, then you create an index template. Whenever you create a
new index and if a template exists that matches it, then the template will
be applied along with whatever you have predefined inside the template. An
example is below (taken directly from the docs -
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-templates.html):
curl -XPUT localhost:9200/_template/template_1 -d '
{
template : te*,
settings : {
number_of_shards : 1
},
mappings : {
type1 : {
_source : { enabled : false }
}
}
}
'
The above template says, if anybody creates a new index in the future with
a name that starts with te (te*), then automatically make 1 shard for it,
create a type named type1, and disable its _source field.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/23313c6a-ac12-49a1-aa57-23e6be854ee1%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: Setting GC for ES
FWIW I hack bin/elasticsearch.in.sh for modifying the JAVA_OPTS there. Jörg On Fri, Feb 14, 2014 at 8:33 PM, Mark Walkom ma...@campaignmonitor.comwrote: I'm a little clueless when it comes to java options in ES, and was wondering where I define things like GC? I did try setting it in ES_JAVA_OPTS under /etc/default/elasticsearch, however when I did ES wouldn't start, so either my syntax is wrong, or something else. Can anyone provide a few pointers for me? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624b7g4YwRDfeHU1OJG9FBHmGO6Zt1UK%3DTd67Wx9K%2BhKq5g%40mail.gmail.com . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHbHyRGFRWN%3DCkyf1TYJZ7SUs0qrfyBfNFJH5m1bSD1kg%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Marvel not collecting system metrics
Tony, We appear to be getting the information back in the cgroup currently; so what's provided is fairly good. It would be better obviously to grab them from the cgroup and push it in; but that would be external to the container (unless you mounted the cgroups in the container) So for my case it works; some spots I do get the 'host' data and not the container data but it's good enough for the moment. Scott On Fri, Feb 14, 2014 at 7:41 AM, Tony Su tonysu...@gmail.com wrote: On Friday, February 14, 2014 4:06:06 AM UTC-8, Binh Ly wrote: May I ask which OS you are running ES on? IIRC Docker is a management tool for LXC. So, it does pique the thoughts... Where should marvel be pulling stats in an LXC deployment? It's not fully isolated like other virtualization technologies and access to various Host processes are blocked. I suppose it might be possible to point to statistics derived by Docker, but that probably requires special configuration. Some thoughts, Tony -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/dhpxaOuoZWI/unsubscribe. To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b409c6f0-1ea3-44f1-bffc-423edc4bb792%40googlegroups.com . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALDJu4XXVEJSdb4zs9sF6%3DgM92UK15K7A%2BUJ%2B8ugBg6Kn_yB_Q%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Setting GC for ES
^^ Same. For the package install, it's under /usr/share/elasticsearch/bin/elasticsearch.in.sh. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9fb3060d-c78a-4f08-be55-da99b275e748%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Knapsack Plugin
Dan, no problem, I can build a version against 0.19.8 Jörg -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGeB%2B9v4icxkv3FK2Orx35Ax1pV8hJZthN94RXdxSPx5A%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Is has_child (parent/child) the best strategy for my use-case involving products and inventory?
argh I just wrote a reply but google ate it apparently. So, are you suggesting that scrapping parent/child and simply storing all of the retailer data in the product document is a safer bet. I imagine we could rate limit our product indexing. However this now gives me two concerns: 1 - the size of the product document. We would have product documents with 20,000+ UPC entries as nested objects! and 2 - would our search results also return those 20,000 nested objects? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/413303b2-a5f2-441e-94da-db7e802eff57%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
