date:20150401

What version are you on?

You can try deleting the snapshot and trying again.

On 1 April 2015 at 21:53, Pradeep Reddy 
wrote:

> I am using cloud aws plugin, Snapshots are taken directly to S3.
> It's a 3 node cluster.
>
> *ES config:*
> cloud:
> aws:
> access_key: xxx
> secret_key: yyy
> protocol: http
> region: us-west-1
> discovery.type: ec2
> discovery.ec2.groups: elasticsearch
> repositories:
> s3:
> bucket: "xxx"
> region: "us-west-1"
>
> When I run this
>
>
>> curl -XGET
>> 'localhost:9200/_snapshot/my_s3_repository/2015.03.11.06.04.39/_status?pretty'
>
>
> Part of the output is
>   "snapshots" : [ {
> "snapshot" : "2015.03.11.06.04.39",
> "repository" : "my_s3_repository",
> "state" : "ABORTED",
> "shards_stats" : {
>   "initializing" : 0,
>   "started" : 0,
>   "finalizing" : 0,
>   "done" : 17,
>   "failed" : 3,
>   "total" : 20
> },
>
> When I run
>
>> curl localhost:9200/_snapshot/_all?pretty
>
> it shows IN_PROGRESS
>  {
> "snapshot" : "2015.03.11.06.04.39",
> "indices" : [ "playground_video", "playground_admin",
> "playground_profile", "playground_feed" ],
> "state" : "IN_PROGRESS",
> "start_time" : "2015-03-11T06:04:39.044Z",
> "start_time_in_millis" : 1426053879044,
> "failures" : [ ],
> "shards" : {
>   "total" : 0,
>   "failed" : 0,
>   "successful" : 0
> }
>   }
>
>  curl -XDELETE
>> 'localhost:9200/_snapshot/my_s3_repository/2015.03.11.06.04.39'
>
> this operation never completes. I left it running for a day..still no use..
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/4a34a807-d596-4ca7-b90b-f709f52d624b%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-b%3DgkVTUrXPnGsLUq7Qrt1_PaF%3Du0hQ6kfi2F7zQDEXA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: CAT API

Total = primary + replica, so it makes sense.

On 2 April 2015 at 14:06, Nishad Karekar  wrote:

> I am unable to understand the results from the CAT API
>
> curl 'http://hdpdncwy0001.global.shareddev.acxiom.net:9200/_cat/indices?v'
>
> health index pri rep docs.count docs.deleted store.size pri.store.size
>
> green  ce  5   1  770000  4.7tb  2.3tb
>
>
>
> From the result I see that the replication is 1 yet my
>
> Total Store size is two times my primary store size.Can someone please
> explain what that is the case?
>
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/d91cef67-30d3-458a-9709-93d298a8d136%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_6pD%3Dfjo4xa0hnh-Px6wo8mJHTvowsB-rgZoAxXLQLhg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Will turnning allow_primary on delete the exist shard?

The docs mention this is a possibility -
http://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-reroute.html

allocate

Allocate an unassigned shard to a node. Accepts the index and shard for
index name and shard number, and node to allocate the shard to. It also
accepts allow_primary flag to explicitly specify that it is allowed to
explicitly allocate a primary shard (might result in data loss).



On 2 April 2015 at 14:39,  wrote:

> Hi,
>
> We had some outage wiht ES cluster. So that we cutoff the incomming and
> outgoing traffic to/from ES cluster.
>
> but we see lots of unassigned shards and get get them allocated
> automatically.
>
> Question is, will tuning allow_primary flag on when manually allocating
> the shards delete the existing data?
> I did so, and saw those shards immediately became available with size 0b.
>
> THanks,
> Yang
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/7ebd6baf-dd5b-40db-8894-a203b9607e8a%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8jTTpObh0B6KCDLoyhtgFMtxNeM4kscAjFbwkYTOU1AQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Will turnning allow_primary on delete the exist shard?

2015-04-01 Thread yang

Hi, 

We had some outage wiht ES cluster. So that we cutoff the incomming and 
outgoing traffic to/from ES cluster.

but we see lots of unassigned shards and get get them allocated 
automatically.

Question is, will tuning allow_primary flag on when manually allocating the 
shards delete the existing data?
I did so, and saw those shards immediately became available with size 0b.

THanks,
Yang

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/7ebd6baf-dd5b-40db-8894-a203b9607e8a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

CAT API

2015-04-01 Thread Nishad Karekar

I am unable to understand the results from the CAT API

curl 'http://hdpdncwy0001.global.shareddev.acxiom.net:9200/_cat/indices?v'

health index pri rep docs.count docs.deleted store.size pri.store.size 

green  ce  5   1  770000  4.7tb  2.3tb 



>From the result I see that the replication is 1 yet my 

Total Store size is two times my primary store size.Can someone please 
explain what that is the case?



-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d91cef67-30d3-458a-9709-93d298a8d136%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

get every Nth doc with sorted result

2015-04-01 Thread Ted Smith

Hello:

I have a query sorting over a field and returns all hits (I set size to max)
Is there any way in query to specify that I only want 
the first, 6th, 11th,  so it would only return these, instead of return 
all  of
them that I have to skip unwanted.

Thanks very much

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/ba3ee392-083d-4ecc-bbff-a6900cff06d3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: 403 from kibana when searching closed indices

You might want to raise an issue on Github for a feature request then as
it's possible to just ignore.

On 2 April 2015 at 09:15, Derry O' Sullivan  wrote:

> Hi Mark,
>
> The reason i asked was that i closed the index due to the fact that i
> didn't want people to query that data temporarily (vs
> deleting/permanently). I thought that Kibana would have handled the closed
> index vs throwing a 403 exception. Without looking at the developer tools
> though, you don't get any errors from kibana - just blank widgets.
>
> Derry
>
> On 1 April 2015 at 22:36, Mark Walkom  wrote:
>
>> This is dangerous as it can lead to you not knowing that data isn't
>> available.
>>
>> If you have it closed and are considering deleting it to resolve this,
>> then why even bother closing it in the first place, just delete it.
>>
>> On 2 April 2015 at 00:37, Derry O' Sullivan  wrote:
>>
>>> Hi all,
>>>
>>> Am using the standard ELK stack and am having an issue with kibana
>>> currently.
>>>
>>> I close one of my indices (e.g. logstash-2015.03.30). When i do a search
>>> in kibana for data involving that index, i get a 403
>>> (IndexClosedException).  I would have expected Kibana/Elasticsearch to know
>>> the index is closed and not to use in in search.
>>>
>>> Is there anyway to tell kibana to ignore that index in search? Would
>>> deleting the index be an alternative?
>>>
>>> Derry
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearch+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elasticsearch/2f6f60d5-fbf0-4ad6-be8e-d4769e62c618%40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "elasticsearch" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/elasticsearch/K7Xg4ZqnpZA/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9xgzJmS%2B8dWSvLMZcHP0ymvgu%3DtapTqniZg-Z6Zse_sQ%40mail.gmail.com
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAOGQnJr%2B_0m3P1UxfWdD8oKhZ%3DH%3DwJ-4CuJPFw-t%2BiLYb1C58g%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_gwoLb3tyf8tQCsN3BJGfTEgz5XXc1rOeCyUMrgBKRKA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: php client benchmarks, which library is faster?

2015-04-01 Thread Zelfapp

Really appreciate you reaching out. We are planning on continuing to use 
es-php for all of the reasons you laid out. Elastica is a great project and 
we started with that, but ultimately decided to work with the officially 
supported project. Our main use for elasticsearch will be for searching an 
index of 25,000 products on an ecommerce site. We'll have to checkout 2.0. 
Thank you.

On Wednesday, April 1, 2015 at 3:27:46 PM UTC-7, zachary.tong wrote:
>
> Heya, author of the official PHP client here.  The only benchmark that I 
> know of is the one you posted, and it is indeed a bit out of date.  Afraid 
> I don't have any hard numbers to give you either, I've never written a 
> benchmark to compare the two clients myself.
>
> There are several considerations when it comes to performance and 
> benchmarking.  What is your end-goal?  Do you want high throughput? Low 
> latency? Both?  For throughput, both Elastica and Elasticsearch-PHP are 
> largely equivalent, once the clients have both been "warmed".  Both clients 
> use PHP's `curl_multi` interface for talking to the server, so the 
> difference in throughput boils down to relatively minor differences in code 
> execution paths.  The majority of the time is dominated by network 
> transfer, so they are roughly equivalent.
>
> Under default settings, Elastica tends to beat Elasticsearch-PHP for 
> latency on one-off queries.  ES-PHP 1.0 uses Guzzle3, which is a bit 
> "bulky".  The first time ES-PHP fires off a query, it needs to autoload a 
> lot of Guzzle classes from disk, which can slow it down considerably 
> compared to Elastica.  If you have a bytecode cache installed, this 
> difference largely disappears.  Or if you use HHVM.
>
> If you switch ES-PHP over to the `CurlMultiConnection` class, the 
> difference largely disappears too, since this effectively replaces Guzzle 
> with a single class that performs the same workload.
>
> Finally, ES-PHP 2.0.beta1 was tagged a month or so ago, and offers a new 
> "async" mode that better leverages `curl_multi`.  It allows you to send 
> requests in parallel to nodes in batches.  This crudely approximates 
> threading in other languages, and can give a huge boost in throughput. 
>  From my (informal, unscientific) testing, it's about 90-170% faster than 
> ES-PHP 1.0 (note this was on a single node, theoretically greater boost for 
> multi-node cluster).  Even then synchronous, non-batch mode was quite a bit 
> faster:
>
>
> 
> I don't have a direct comparison against Elastica, but I imagine it will 
> be faster for the same reason it is faster than ES-PHP 1.0 ... it is simply 
> executing in parallel.
>
> Other benefits of the official library include someone (me) who is paid to 
> maintain it, and develop new features like the async functionality in 2.0. 
>  ES-PHP also undergoes continuous integration testing against multiple ES 
> versions, using our extensive test suite (which all the clients run).  That 
> means we support the *entire* set Elasticsearch APIs, and support new 
> APIs shortly after they are added.  It supports connection pooling, 
> configurable selection strategies, etc. It's highly pluggable, so you can 
> write your own behavior if you want.  And finally, it has similar syntax to 
> other language clients, in case you work in a multi-language environment.
>
> Now, with all that said, Elastica is a great project.  Ruflin has done a 
> great job maintaining it over the years, and a lot of people prefer the API 
> to the more generic / verbose ES-PHP API.  I've told people before that if 
> they are happy with Elastica, there isn't a suuuper compelling reason to 
> switch.  The new 2.0 async behavior may be a reason, but the jury is still 
> out on that.
>
> Cheers,
> -Zach
>
>
>
> On Wednesday, April 1, 2015 at 5:55:58 PM UTC-4, Zelfapp wrote:
>>
>> The only php client benchmark I can find for elasticsearch seems big time 
>> outdated: http://jolicode.com/blog/elasticsearch-php-clients-test-drive
>>
>> We are currently developing using the elasticsearch-php official library 
>> https://github.com/elastic/elasticsearch-php
>>
>> The benchmark on the jolicode.com sight makes it appear that elastica 
>> has the official php library beat, but this is outdated benchmark. 
>> Nervetattoo is no longer being developed for example. We don't have the 
>> time to create our own benchmark. 
>>
>> Hoping someone out there can tell me the benchmark status of the official 
>> library compared to elastica, which is being actively developed.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/edafb50d-986d-4133-9039-24eb4e

Re: Shard Allocation Problem

It looks like this is only happening for replicas. I'd suggest picking a
single index, remove the replicas and then re-add them and see if they are
allocated successfully.

On 2 April 2015 at 05:45, Marcelo Paes Rech 
wrote:

> The ElasticSearch Team closed the issue, but I didn't have an answer. I
> still looking for a solution.
>
>
> Regards.
>
> Em sábado, 28 de março de 2015 16:29:05 UTC-3, Marcelo Paes Rech escreveu:
>>
>> Hi guys,
>>
>> Our cluster has 7 data nodes. Last week one of them was turned off. We
>> restarted the node and then the cluster started rebalancing. The problem is
>> it never finishes rebalancing.
>>
>> Node logs shows an update coming from master node during the rebalancing,
>> seconds later the rebalancing is canceled and then it starts again.
>>
>> I have created #10281
>>  on GitHub, but I
>> am including the problem here too. Maybe somebody ran into this problem and
>> already fixed it.
>>
>> The logs were collected in different moments but they represent the same
>> behaviour.
>>
>> The log follows:
>>
>> Data Node
>>
>> https://gist.github.com/marcelopaesrech/7b9f94310ff7a24bed9e
>>
>> Master Node
>>
>> https://gist.github.com/marcelopaesrech/c80542a85c83d9c1756d
>>
>>
>> Regards.
>>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/d7e23198-fd2a-4b41-aeed-cf9b8cc23ae7%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-RicbxJ_azuKDPPjTX11tX2baXYRQpJqNBvgHTPtC1bg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: ES version and phonetic version

2015-04-01 Thread David Pilato

Do you have the same issue when using 2.2.0?
What can you see in logs?

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

> Le 2 avr. 2015 à 00:05, Mark Bramer  a écrit :
> 
> 
> No.  The fact that there's not a 2.2.1 on the main page of the phonetic 
> GitHub page (https://github.com/elastic/elasticsearch-analysis-phonetic) made 
> me think that the 2.2.1 I have must have been a modified 2.2.0.  But, upon 
> clicking around, I found out that there is a 2.2.1 version which is a 
> snapshot 
> (https://github.com/elastic/elasticsearch-analysis-phonetic/tree/es-1.2). I'm 
> deducing that this is the one I have. The prior developer must have cloned 
> this and built it in-house.
> 
>> On Wednesday, April 1, 2015 at 5:55:48 PM UTC-4, David Pilato wrote:
>> Sorry. I don't follow. You are not using official plugin version 2.2.0 but a 
>> modified version?
>> 
>> David
>> 
>>> Le 1 avr. 2015 à 23:42, Mark Bramer  a écrit :
>>> 
>>> 
>>> There's a 2.2.1 snapshot.  I don't believe the one I'm using was modified 
>>> from this one.
>>> 
 On Wednesday, April 1, 2015 at 5:32:40 PM UTC-4, Mark Bramer wrote:
 I inherited this from someone else (how many times have we all heard 
 that??) who named the file I'm using - perhaps he made some changes and 
 rebuilt...?  Either way, I'm using ES 1.2 and phonetic 2.1 which according 
 to the matrix, should work...
 
> On Wednesday, April 1, 2015 at 5:28:22 PM UTC-4, David Pilato wrote:
> Is there a version 2.2.1?
> 
> --
> David ;-)
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> 
>> Le 1 avr. 2015 à 23:24, Mark Bramer  a écrit :
>> 
>> 
>> Upon second review of the matrix, I realized that I AM in fact pairing a 
>> supported version pair of ES 1.2.2 and phonetic 2.2.1.  So although my 
>> version-matching questions are answered, I'm still stuck with why ES 
>> won't start after plugin install.  I'm guessing that will require more 
>> details from the log...
>> 
>> 
>>> On Wednesday, April 1, 2015 at 5:20:42 PM UTC-4, Mark Bramer wrote:
>>> 
>>> I looked again at the plugin GitHub site and I now see the matrix 
>>> (below) I should have seen before posting this.  My sincere apologies 
>>> for not doing my homework, but I'm still interested in hearing any 
>>> relevant details that may have popped into anyone's head while reading 
>>> this.
>>> 
>>> elasticsearch   Phonetic Analysis PluginDocs
>>> master  Build from source   See below
>>> es-1.x  Build from source   2.6.0-SNAPSHOT
>>> es-1.5  2.5.0   2.5.0
>>> es-1.4  2.4.2   2.4.2
>>> < 1.4.3 2.4.1   2.4.1
>>> es-1.3  2.3.0   2.3.0
>>> es-1.2  2.2.0   2.2.0
>>> es-1.1  2.1.0   2.1.0
>>> es-1.0  2.0.0   2.0.0
>>> es-0.90 1.8.0   1.8.0
>>> 
 On Wednesday, April 1, 2015 at 5:17:42 PM UTC-4, Mark Bramer wrote:
 Hello,
 
 In general, how important is it that plugin versions match the 
 Elasticsearch version?  Or are (can) plugins be versioned with no 
 correlation to Elasticsearch's version at all?  Specific case:
 
 I have ES 1.2.2 and phonetic plugin 2.2.1.  In good practice, should I 
 be using ES 2.2.1?  Or vice versa; should I use phonetic 1.2.2?  so 
 that the versions match?
 
 I was able to install the plugin on a 3-machine cluster using "plugin 
 --url file:///home/username/elasticsearch-analysis-phonetic-2.2.1 
 --install phonetic-plugin" on all 3 machines (this may not be EXACTLY 
 correct command, I'm going by memory).  On all 3 machines, this 
 created the 'plugins' directory in my ES home and appeared to properly 
 place the plugin in the directory. I stopped ES using "service 
 elasticsearch stop".  I tried restarting using "... start" and ES will 
 no longer start.  I see errors in the logs that seem to pertain to 
 plugins.  Given this, plus the fact ES worked fine before the plugin 
 installation, i uninstall the plugin using "plugin --remove 
 phonetic-plugin".  This removed the plugin from the plugins directory 
 but ES still won't start. The logs still contain plugin-related 
 errors.  Upon looking, I see the plugins directory is still present.  
 I manually delete the directory, and now ES starts perfectly fine.
 
 My main question is: Should my first course of troubleshooting be 
 obtaining a 1.2.2 version of the phonetic plugin?  Or is something 
 else likely the culprit?
 
 *Some notes: 
 1. This is a disconnected environment with no internet access, so I 
 will always have to do 'manual' installs.  
 2. I currently do not have the exact errors to include here, but I can 
 hand-write some error details tomorrow when I'm b

[ANN] Deprecating Rivers

Hi All,
We've recently announced that Rivers in Elasticsearch are actively
deprecated as of the 1.5 release, the 2.0 release will still maintain that
functionality to ease migration, but we recommend you start to leverage the
official clients to replace any Rivers you may have in use.

If you have feedback around this process then feel free to add a comment to
this Github issue - https://github.com/elastic/elasticsearch/issues/10345

And for more information please see this blog post on our site -
https://www.elastic.co/blog/deprecating_rivers

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X90gyaxCnV9ygkdEAJirv_4p8F3E_1J9jQU_axn9MBCGw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: php client benchmarks, which library is faster?

2015-04-01 Thread zachary.tong

Heya, author of the official PHP client here.  The only benchmark that I 
know of is the one you posted, and it is indeed a bit out of date.  Afraid 
I don't have any hard numbers to give you either, I've never written a 
benchmark to compare the two clients myself.

There are several considerations when it comes to performance and 
benchmarking.  What is your end-goal?  Do you want high throughput? Low 
latency? Both?  For throughput, both Elastica and Elasticsearch-PHP are 
largely equivalent, once the clients have both been "warmed".  Both clients 
use PHP's `curl_multi` interface for talking to the server, so the 
difference in throughput boils down to relatively minor differences in code 
execution paths.  The majority of the time is dominated by network 
transfer, so they are roughly equivalent.

Under default settings, Elastica tends to beat Elasticsearch-PHP for 
latency on one-off queries.  ES-PHP 1.0 uses Guzzle3, which is a bit 
"bulky".  The first time ES-PHP fires off a query, it needs to autoload a 
lot of Guzzle classes from disk, which can slow it down considerably 
compared to Elastica.  If you have a bytecode cache installed, this 
difference largely disappears.  Or if you use HHVM.

If you switch ES-PHP over to the `CurlMultiConnection` class, the 
difference largely disappears too, since this effectively replaces Guzzle 
with a single class that performs the same workload.

Finally, ES-PHP 2.0.beta1 was tagged a month or so ago, and offers a new 
"async" mode that better leverages `curl_multi`.  It allows you to send 
requests in parallel to nodes in batches.  This crudely approximates 
threading in other languages, and can give a huge boost in throughput. 
 From my (informal, unscientific) testing, it's about 90-170% faster than 
ES-PHP 1.0 (note this was on a single node, theoretically greater boost for 
multi-node cluster).  Even then synchronous, non-batch mode was quite a bit 
faster:

I don't have a direct comparison against Elastica, but I imagine it will be 
faster for the same reason it is faster than ES-PHP 1.0 ... it is simply 
executing in parallel.

Other benefits of the official library include someone (me) who is paid to 
maintain it, and develop new features like the async functionality in 2.0. 
 ES-PHP also undergoes continuous integration testing against multiple ES 
versions, using our extensive test suite (which all the clients run).  That 
means we support the *entire* set Elasticsearch APIs, and support new APIs 
shortly after they are added.  It supports connection pooling, configurable 
selection strategies, etc. It's highly pluggable, so you can write your own 
behavior if you want.  And finally, it has similar syntax to other language 
clients, in case you work in a multi-language environment.

Now, with all that said, Elastica is a great project.  Ruflin has done a 
great job maintaining it over the years, and a lot of people prefer the API 
to the more generic / verbose ES-PHP API.  I've told people before that if 
they are happy with Elastica, there isn't a suuuper compelling reason to 
switch.  The new 2.0 async behavior may be a reason, but the jury is still 
out on that.

Cheers,
-Zach

On Wednesday, April 1, 2015 at 5:55:58 PM UTC-4, Zelfapp wrote:
>
> The only php client benchmark I can find for elasticsearch seems big time 
> outdated: http://jolicode.com/blog/elasticsearch-php-clients-test-drive
>
> We are currently developing using the elasticsearch-php official library 
> https://github.com/elastic/elasticsearch-php
>
> The benchmark on the jolicode.com sight makes it appear that elastica has 
> the official php library beat, but this is outdated benchmark. Nervetattoo 
> is no longer being developed for example. We don't have the time to create 
> our own benchmark. 
>
> Hoping someone out there can tell me the benchmark status of the official 
> library compared to elastica, which is being actively developed.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/862de673-9ea2-4216-9284-5084e8bf716a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: 403 from kibana when searching closed indices

2015-04-01 Thread Derry O' Sullivan

Hi Mark,

The reason i asked was that i closed the index due to the fact that i
didn't want people to query that data temporarily (vs
deleting/permanently). I thought that Kibana would have handled the closed
index vs throwing a 403 exception. Without looking at the developer tools
though, you don't get any errors from kibana - just blank widgets.

Derry

On 1 April 2015 at 22:36, Mark Walkom  wrote:

> This is dangerous as it can lead to you not knowing that data isn't
> available.
>
> If you have it closed and are considering deleting it to resolve this,
> then why even bother closing it in the first place, just delete it.
>
> On 2 April 2015 at 00:37, Derry O' Sullivan  wrote:
>
>> Hi all,
>>
>> Am using the standard ELK stack and am having an issue with kibana
>> currently.
>>
>> I close one of my indices (e.g. logstash-2015.03.30). When i do a search
>> in kibana for data involving that index, i get a 403
>> (IndexClosedException).  I would have expected Kibana/Elasticsearch to know
>> the index is closed and not to use in in search.
>>
>> Is there anyway to tell kibana to ignore that index in search? Would
>> deleting the index be an alternative?
>>
>> Derry
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/2f6f60d5-fbf0-4ad6-be8e-d4769e62c618%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/K7Xg4ZqnpZA/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9xgzJmS%2B8dWSvLMZcHP0ymvgu%3DtapTqniZg-Z6Zse_sQ%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAOGQnJr%2B_0m3P1UxfWdD8oKhZ%3DH%3DwJ-4CuJPFw-t%2BiLYb1C58g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to perform Sum of a field to a group by data....

2015-04-01 Thread Adrien Grand

Your request looks good. Did you maybe mis-type the field name for the sum
aggregation?

On Wed, Apr 1, 2015 at 5:06 PM, Muddadi Hemaanusha <
hemaanusha.bu...@gmail.com> wrote:

> Hi Team,
>
> I would like to get the exact result of an SQL query
>
> select sum(Amount) from Emp group by (Name) Here If I take an example
> like
>
> id  Name   amount
> 1   A1100
> 2   A2 200
> 3   A3 300
> 4   A1 200
> 5   A2 300
>
> So for the above query the result will be Sum(Amount) by grouping the
> names A1, A2, A3 and for A1--> sum(100+200=300), A2-->sum(200+300=500),
> A3-->sum(300)...
>
> For this query I have used the following equivalent query using
> Aggregations as shown below:
>
> GET ABCD/_search
> {
> "size": 0,
>
>   "aggs": {
> "group_trans": {
>   "terms": {
> "field": "trans"
>   },
>   "aggs": {
> "sum_ordamt":{
> "sum":{
> "field":"ordamt"
> }
> }
> }
>   }
> }
>   }
>
> Am getting the result as:
>
> "aggregations": {
>   "group_trans": {
>  "doc_count_error_upper_bound": 81,
>  "sum_other_doc_count": 1371988,
>  "buckets": [
> {
>"key": "7ce2d738-5549-4da2-aed0-db939f3a9188",
>"doc_count": 282,
>"sum_ordamt": {
>   "value": 0
>}
> },
> {
>"key": "8f4b187a-f321-4d25-bbc0-a9e78f5810b8",
>"doc_count": 230,
>"sum_ordamt": {
>   "value": 0
>}
> },
> {
>"key": "ca54db4c-620b-4650-bef6-350e83a0ca2c",
>"doc_count": 170,
>"sum_ordamt": {
>   "value": 0
>}
> }
> .
> .
> .
>  ]
>   }
>}
> }   .Here Am getting sum
> value==0, May I know is there any way to Make a query relevant to the SQL
> query given above for to give the sum depending on the group by names...
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/ab20bcf5-d573-431b-bb9e-7c1207109988%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Adrien

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAO5%3DkAhu6roECh%2BkkPROFqYWNoD6B_nvn-79AEGDD6-FwcvAtA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: ES version and phonetic version


No.  The fact that there's not a 2.2.1 on the main page of the phonetic 
GitHub page (https://github.com/elastic/elasticsearch-analysis-phonetic) 
made me think that the 2.2.1 I have must have been a modified 2.2.0.  But, 
upon clicking around, I found out that there is a 2.2.1 version which is a 
snapshot 
(https://github.com/elastic/elasticsearch-analysis-phonetic/tree/es-1.2). 
I'm deducing that this is the one I have. The prior developer must have 
cloned this and built it in-house.

On Wednesday, April 1, 2015 at 5:55:48 PM UTC-4, David Pilato wrote:
>
> Sorry. I don't follow. You are not using official plugin version 2.2.0 but 
> a modified version?
>
> David
>
> Le 1 avr. 2015 à 23:42, Mark Bramer > a 
> écrit :
>
>
> There's a 2.2.1 snapshot.  I don't believe the one I'm using was modified 
> from this one.
>
> On Wednesday, April 1, 2015 at 5:32:40 PM UTC-4, Mark Bramer wrote:
>>
>> I inherited this from someone else (how many times have we all heard 
>> that??) who named the file I'm using - perhaps he made some changes and 
>> rebuilt...?  Either way, I'm using ES 1.2 and phonetic 2.1 which according 
>> to the matrix, should work...
>>
>> On Wednesday, April 1, 2015 at 5:28:22 PM UTC-4, David Pilato wrote:
>>>
>>> Is there a version 2.2.1?
>>>
>>> --
>>> David ;-)
>>> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>>>
>>> Le 1 avr. 2015 à 23:24, Mark Bramer  a écrit :
>>>
>>>
>>> Upon second review of the matrix, I realized that I AM in fact pairing a 
>>> supported version pair of ES 1.2.2 and phonetic 2.2.1.  So although my 
>>> version-matching questions are answered, I'm still stuck with why ES won't 
>>> start after plugin install.  I'm guessing that will require more details 
>>> from the log...
>>>
>>>
>>> On Wednesday, April 1, 2015 at 5:20:42 PM UTC-4, Mark Bramer wrote:


 I looked again at the plugin GitHub site and I now see the matrix 
 (below) I should have seen before posting this.  My sincere apologies for 
 not doing my homework, but I'm still interested in hearing any relevant 
 details that may have popped into anyone's head while reading this.

 elasticsearchPhonetic Analysis PluginDocsmasterBuild from sourceSee 
 belowes-1.xBuild from source2.6.0-SNAPSHOT 
 
 es-1.52.5.02.5.0 
 
 es-1.42.4.22.4.2 
 <
  
 1.4.32.4.12.4.1 
 
 es-1.32.3.02.3.0 
 
 es-1.22.2.02.2.0 
 
 es-1.12.1.02.1.0 
 
 es-1.02.0.02.0.0 
 
 es-0.901.8.01.8.0 
 
 On Wednesday, April 1, 2015 at 5:17:42 PM UTC-4, Mark Bramer wrote:
>
> Hello,
>
> In general, how important is it that plugin versions match the 
> Elasticsearch version?  Or are (can) plugins be versioned with no 
> correlation to Elasticsearch's version at all?  Specific case:
>
> I have ES 1.2.2 and phonetic plugin 2.2.1.  In good practice, should I 
> be using ES 2.2.1?  Or vice versa; should I use phonetic 1.2.2?  so that 
> the versions match?
>
> I was able to install the plugin on a 3-machine cluster using "plugin 
> --url file:///home/username/elasticsearch-analysis-phonetic-2.2.1 
> --install 
> phonetic-plugin" on all 3 machines (this may not be EXACTLY correct 
> command, I'm going by memory).  On all 3 machines, this created the '
> plugins' directory in my ES home and appeared to properly place the 
> plugin in the directory. I stopped ES using "service elasticsearch 
> stop".  I tried restarting using "... start" and ES will no longer 
> start.  I see errors in the logs that seem to pertain to plugins.  Given 
> this, plus the fact ES worked fine before the plugin installation, i 
> uninstall the plugin using "plugin --remove phonetic-plugin".  This 
> removed the plugin from the plugins directory but ES still won't start. 
> The 
> logs still contain plugin-related errors.  Upon looking, I see the 
> plugins

php client benchmarks, which library is faster?

2015-04-01 Thread Zelfapp

The only php client benchmark I can find for elasticsearch seems big time 
outdated: http://jolicode.com/blog/elasticsearch-php-clients-test-drive

We are currently developing using the elasticsearch-php official library 
https://github.com/elastic/elasticsearch-php

The benchmark on the jolicode.com sight makes it appear that elastica has 
the official php library beat, but this is outdated benchmark. Nervetattoo 
is no longer being developed for example. We don't have the time to create 
our own benchmark. 

Hoping someone out there can tell me the benchmark status of the official 
library compared to elastica, which is being actively developed.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5858b1c7-fb5b-45fe-8feb-86e857c66cf3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ES version and phonetic version

2015-04-01 Thread David Pilato

Sorry. I don't follow. You are not using official plugin version 2.2.0 but a 
modified version?

David

> Le 1 avr. 2015 à 23:42, Mark Bramer  a écrit :
> 
> 
> There's a 2.2.1 snapshot.  I don't believe the one I'm using was modified 
> from this one.
> 
>> On Wednesday, April 1, 2015 at 5:32:40 PM UTC-4, Mark Bramer wrote:
>> I inherited this from someone else (how many times have we all heard that??) 
>> who named the file I'm using - perhaps he made some changes and rebuilt...?  
>> Either way, I'm using ES 1.2 and phonetic 2.1 which according to the matrix, 
>> should work...
>> 
>>> On Wednesday, April 1, 2015 at 5:28:22 PM UTC-4, David Pilato wrote:
>>> Is there a version 2.2.1?
>>> 
>>> --
>>> David ;-)
>>> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>>> 
 Le 1 avr. 2015 à 23:24, Mark Bramer  a écrit :
 
 
 Upon second review of the matrix, I realized that I AM in fact pairing a 
 supported version pair of ES 1.2.2 and phonetic 2.2.1.  So although my 
 version-matching questions are answered, I'm still stuck with why ES won't 
 start after plugin install.  I'm guessing that will require more details 
 from the log...
 
 
> On Wednesday, April 1, 2015 at 5:20:42 PM UTC-4, Mark Bramer wrote:
> 
> I looked again at the plugin GitHub site and I now see the matrix (below) 
> I should have seen before posting this.  My sincere apologies for not 
> doing my homework, but I'm still interested in hearing any relevant 
> details that may have popped into anyone's head while reading this.
> 
> elasticsearch Phonetic Analysis PluginDocs
> masterBuild from source   See below
> es-1.xBuild from source   2.6.0-SNAPSHOT
> es-1.52.5.0   2.5.0
> es-1.42.4.2   2.4.2
> < 1.4.3   2.4.1   2.4.1
> es-1.32.3.0   2.3.0
> es-1.22.2.0   2.2.0
> es-1.12.1.0   2.1.0
> es-1.02.0.0   2.0.0
> es-0.90   1.8.0   1.8.0
> 
>> On Wednesday, April 1, 2015 at 5:17:42 PM UTC-4, Mark Bramer wrote:
>> Hello,
>> 
>> In general, how important is it that plugin versions match the 
>> Elasticsearch version?  Or are (can) plugins be versioned with no 
>> correlation to Elasticsearch's version at all?  Specific case:
>> 
>> I have ES 1.2.2 and phonetic plugin 2.2.1.  In good practice, should I 
>> be using ES 2.2.1?  Or vice versa; should I use phonetic 1.2.2?  so that 
>> the versions match?
>> 
>> I was able to install the plugin on a 3-machine cluster using "plugin 
>> --url file:///home/username/elasticsearch-analysis-phonetic-2.2.1 
>> --install phonetic-plugin" on all 3 machines (this may not be EXACTLY 
>> correct command, I'm going by memory).  On all 3 machines, this created 
>> the 'plugins' directory in my ES home and appeared to properly place the 
>> plugin in the directory. I stopped ES using "service elasticsearch 
>> stop".  I tried restarting using "... start" and ES will no longer 
>> start.  I see errors in the logs that seem to pertain to plugins.  Given 
>> this, plus the fact ES worked fine before the plugin installation, i 
>> uninstall the plugin using "plugin --remove phonetic-plugin".  This 
>> removed the plugin from the plugins directory but ES still won't start. 
>> The logs still contain plugin-related errors.  Upon looking, I see the 
>> plugins directory is still present.  I manually delete the directory, 
>> and now ES starts perfectly fine.
>> 
>> My main question is: Should my first course of troubleshooting be 
>> obtaining a 1.2.2 version of the phonetic plugin?  Or is something else 
>> likely the culprit?
>> 
>> *Some notes: 
>> 1. This is a disconnected environment with no internet access, so I will 
>> always have to do 'manual' installs.  
>> 2. I currently do not have the exact errors to include here, but I can 
>> hand-write some error details tomorrow when I'm back on the job if you 
>> want more info.
>> 
>> Thanks, 
>> Mark
 
 -- 
 You received this message because you are subscribed to the Google Groups 
 "elasticsearch" group.
 To unsubscribe from this group and stop receiving emails from it, send an 
 email to elasticsearc...@googlegroups.com.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/elasticsearch/879ec193-e550-48cc-af6d-9944c0904087%40googlegroups.com.
 For more options, visit https://groups.google.com/d/optout.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/e13e4f80-504c-430e-b05e-b4f0e0a9c797%40googlegroups.co

Re: ES version and phonetic version


There's a 2.2.1 snapshot.  I don't believe the one I'm using was modified 
from this one.

On Wednesday, April 1, 2015 at 5:32:40 PM UTC-4, Mark Bramer wrote:
>
> I inherited this from someone else (how many times have we all heard 
> that??) who named the file I'm using - perhaps he made some changes and 
> rebuilt...?  Either way, I'm using ES 1.2 and phonetic 2.1 which according 
> to the matrix, should work...
>
> On Wednesday, April 1, 2015 at 5:28:22 PM UTC-4, David Pilato wrote:
>>
>> Is there a version 2.2.1?
>>
>> --
>> David ;-)
>> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>>
>> Le 1 avr. 2015 à 23:24, Mark Bramer  a écrit :
>>
>>
>> Upon second review of the matrix, I realized that I AM in fact pairing a 
>> supported version pair of ES 1.2.2 and phonetic 2.2.1.  So although my 
>> version-matching questions are answered, I'm still stuck with why ES won't 
>> start after plugin install.  I'm guessing that will require more details 
>> from the log...
>>
>>
>> On Wednesday, April 1, 2015 at 5:20:42 PM UTC-4, Mark Bramer wrote:
>>>
>>>
>>> I looked again at the plugin GitHub site and I now see the matrix 
>>> (below) I should have seen before posting this.  My sincere apologies for 
>>> not doing my homework, but I'm still interested in hearing any relevant 
>>> details that may have popped into anyone's head while reading this.
>>>
>>> elasticsearchPhonetic Analysis PluginDocsmasterBuild from sourceSee 
>>> belowes-1.xBuild from source2.6.0-SNAPSHOT 
>>> 
>>> es-1.52.5.02.5.0 
>>> 
>>> es-1.42.4.22.4.2 
>>> <
>>>  
>>> 1.4.32.4.12.4.1 
>>> 
>>> es-1.32.3.02.3.0 
>>> 
>>> es-1.22.2.02.2.0 
>>> 
>>> es-1.12.1.02.1.0 
>>> 
>>> es-1.02.0.02.0.0 
>>> 
>>> es-0.901.8.01.8.0 
>>> 
>>> On Wednesday, April 1, 2015 at 5:17:42 PM UTC-4, Mark Bramer wrote:

 Hello,

 In general, how important is it that plugin versions match the 
 Elasticsearch version?  Or are (can) plugins be versioned with no 
 correlation to Elasticsearch's version at all?  Specific case:

 I have ES 1.2.2 and phonetic plugin 2.2.1.  In good practice, should I 
 be using ES 2.2.1?  Or vice versa; should I use phonetic 1.2.2?  so that 
 the versions match?

 I was able to install the plugin on a 3-machine cluster using "plugin 
 --url file:///home/username/elasticsearch-analysis-phonetic-2.2.1 
 --install 
 phonetic-plugin" on all 3 machines (this may not be EXACTLY correct 
 command, I'm going by memory).  On all 3 machines, this created the '
 plugins' directory in my ES home and appeared to properly place the 
 plugin in the directory. I stopped ES using "service elasticsearch stop". 
  I tried restarting using "... start" and ES will no longer start.  I 
 see errors in the logs that seem to pertain to plugins.  Given this, plus 
 the fact ES worked fine before the plugin installation, i uninstall the 
 plugin using "plugin --remove phonetic-plugin".  This removed the 
 plugin from the plugins directory but ES still won't start. The logs still 
 contain plugin-related errors.  Upon looking, I see the plugins directory 
 is still present.  I manually delete the directory, and now ES starts 
 perfectly fine.

 My main question is: Should my first course of troubleshooting be 
 obtaining a 1.2.2 version of the phonetic plugin?  Or is something else 
 likely the culprit?

 *Some notes: 
 1. This is a disconnected environment with no internet access, so I 
 will always have to do 'manual' installs.  
 2. I currently do not have the exact errors to include here, but I can 
 hand-write some error details tomorrow when I'm back on the job if you 
 want 
 more info.

 Thanks, 
 Mark

>>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it

Re: 403 from kibana when searching closed indices

This is dangerous as it can lead to you not knowing that data isn't
available.

If you have it closed and are considering deleting it to resolve this, then
why even bother closing it in the first place, just delete it.

On 2 April 2015 at 00:37, Derry O' Sullivan  wrote:

> Hi all,
>
> Am using the standard ELK stack and am having an issue with kibana
> currently.
>
> I close one of my indices (e.g. logstash-2015.03.30). When i do a search
> in kibana for data involving that index, i get a 403
> (IndexClosedException).  I would have expected Kibana/Elasticsearch to know
> the index is closed and not to use in in search.
>
> Is there anyway to tell kibana to ignore that index in search? Would
> deleting the index be an alternative?
>
> Derry
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/2f6f60d5-fbf0-4ad6-be8e-d4769e62c618%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9xgzJmS%2B8dWSvLMZcHP0ymvgu%3DtapTqniZg-Z6Zse_sQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Reading from an alias which has two indexes

It'd probably return both, depends on what your query is.

On 31 March 2015 at 04:49, Kumar S  wrote:

> Hi,
> If i have two indexes under an alias say "Old Index" and "New Index". the
> old index contains a doc identified by #doc1, version 1 and the i start
> populating "new index" (and stop writing to old index). I then index a
> updated version of #doc 1 to "New Index". Now "New Index" contains #doc1,
> version 2.
>
> Now, when i issue a read query to the alias (which ahs Old & New Index
> under it), which version of the doc would it return, #doc 1, version 2?
>
> Thanks for your prompt response.
>
> Kumar Subramanian.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/a2fecda7-2345-4a92-a94d-a68982bf4bd7%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8OO-zV0G6nw%2BXf%2Bd171CYf2pwWF4oUow%2BFrjYta_5HMA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: ES version and phonetic version

I inherited this from someone else (how many times have we all heard 
that??) who named the file I'm using - perhaps he made some changes and 
rebuilt...?  Either way, I'm using ES 1.2 and phonetic 2.1 which according 
to the matrix, should work...

On Wednesday, April 1, 2015 at 5:28:22 PM UTC-4, David Pilato wrote:
>
> Is there a version 2.2.1?
>
> --
> David ;-)
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>
> Le 1 avr. 2015 à 23:24, Mark Bramer > a 
> écrit :
>
>
> Upon second review of the matrix, I realized that I AM in fact pairing a 
> supported version pair of ES 1.2.2 and phonetic 2.2.1.  So although my 
> version-matching questions are answered, I'm still stuck with why ES won't 
> start after plugin install.  I'm guessing that will require more details 
> from the log...
>
>
> On Wednesday, April 1, 2015 at 5:20:42 PM UTC-4, Mark Bramer wrote:
>>
>>
>> I looked again at the plugin GitHub site and I now see the matrix (below) 
>> I should have seen before posting this.  My sincere apologies for not doing 
>> my homework, but I'm still interested in hearing any relevant details that 
>> may have popped into anyone's head while reading this.
>>
>> elasticsearchPhonetic Analysis PluginDocsmasterBuild from sourceSee below
>> es-1.xBuild from source2.6.0-SNAPSHOT 
>> 
>> es-1.52.5.02.5.0 
>> 
>> es-1.42.4.22.4.2 
>> <
>>  
>> 1.4.32.4.12.4.1 
>> 
>> es-1.32.3.02.3.0 
>> 
>> es-1.22.2.02.2.0 
>> 
>> es-1.12.1.02.1.0 
>> 
>> es-1.02.0.02.0.0 
>> 
>> es-0.901.8.01.8.0 
>> 
>> On Wednesday, April 1, 2015 at 5:17:42 PM UTC-4, Mark Bramer wrote:
>>>
>>> Hello,
>>>
>>> In general, how important is it that plugin versions match the 
>>> Elasticsearch version?  Or are (can) plugins be versioned with no 
>>> correlation to Elasticsearch's version at all?  Specific case:
>>>
>>> I have ES 1.2.2 and phonetic plugin 2.2.1.  In good practice, should I 
>>> be using ES 2.2.1?  Or vice versa; should I use phonetic 1.2.2?  so that 
>>> the versions match?
>>>
>>> I was able to install the plugin on a 3-machine cluster using "plugin 
>>> --url file:///home/username/elasticsearch-analysis-phonetic-2.2.1 --install 
>>> phonetic-plugin" on all 3 machines (this may not be EXACTLY correct 
>>> command, I'm going by memory).  On all 3 machines, this created the '
>>> plugins' directory in my ES home and appeared to properly place the 
>>> plugin in the directory. I stopped ES using "service elasticsearch stop". 
>>>  I tried restarting using "... start" and ES will no longer start.  I 
>>> see errors in the logs that seem to pertain to plugins.  Given this, plus 
>>> the fact ES worked fine before the plugin installation, i uninstall the 
>>> plugin using "plugin --remove phonetic-plugin".  This removed the 
>>> plugin from the plugins directory but ES still won't start. The logs still 
>>> contain plugin-related errors.  Upon looking, I see the plugins directory 
>>> is still present.  I manually delete the directory, and now ES starts 
>>> perfectly fine.
>>>
>>> My main question is: Should my first course of troubleshooting be 
>>> obtaining a 1.2.2 version of the phonetic plugin?  Or is something else 
>>> likely the culprit?
>>>
>>> *Some notes: 
>>> 1. This is a disconnected environment with no internet access, so I will 
>>> always have to do 'manual' installs.  
>>> 2. I currently do not have the exact errors to include here, but I can 
>>> hand-write some error details tomorrow when I'm back on the job if you want 
>>> more info.
>>>
>>> Thanks, 
>>> Mark
>>>
>>  -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearc...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/879ec193-e550-48cc-af6d-9944c0904087%40googlegroups.com
>  
>

Re: How to enable Garbage Collector logging in an elastic search nodes..

It's not recommended to alter the .in.sh, this file can change between
releases and your alterations can either be lost or can cause other
problems.

I'd suggest you do what Gavin mentioned.

On 1 April 2015 at 18:08, Anil Karaka  wrote:

> Thanks,
>
> I just kept those GC logging options outside the if block.. So that I
> don't have to set that variable, and enabled GC logging by default.
>
> this is in the file /usr/share/elasticsearch/bin/elasticsearch.in.sh
>
> # GC logging options
> if [ "x$ES_USE_GC_LOGGING" != "x" ]; then
>   JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails"
>   JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCTimeStamps"
>   JAVA_OPTS="$JAVA_OPTS -XX:+PrintClassHistogram"
>   JAVA_OPTS="$JAVA_OPTS -XX:+PrintTenuringDistribution"
>   JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCApplicationStoppedTime"
>   JAVA_OPTS="$JAVA_OPTS -Xloggc:/var/log/elasticsearch/gc.log"
> fi
> # Enable gc logging by default
> JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails"
> JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCTimeStamps"
> JAVA_OPTS="$JAVA_OPTS -XX:+PrintClassHistogram"
> JAVA_OPTS="$JAVA_OPTS -XX:+PrintTenuringDistribution"
> JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCApplicationStoppedTime"
> JAVA_OPTS="$JAVA_OPTS -Xloggc:/var/log/elasticsearch/gc.log"
>
>
> On Saturday, March 14, 2015 at 7:18:29 AM UTC+5:30, Gavin Seng wrote:
>>
>>
>> Anil, you can try this:
>>
>> # /etc/sysconfig/elasticsearch
>>
>> export ES_USE_GC_LOGGING=1
>> ES_JAVA_OPTS="-verbose:gc -XX:+PrintGCDateStamps
>> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M
>> -Xloggc:"
>>
>> Gavin
>>
>> On Wednesday, March 4, 2015 at 9:22:50 AM UTC-5, Anil Karaka wrote:
>>>
>>> Can you please tell me what exactly do I need to add in
>>> /etc/default/elasticsearch?
>>>
>>> What should I set ES_USE_GC_LOGGING to?
>>>
>>> I added below line in my dev ES node, and I didn't see the gc.log file
>>> in my log folder.. I want to be sure what to do before changing anything in
>>> my production cluster..
>>>
>>> ES_USE_GC_LOGGING=true
>>>
>>> And what are the GC logging options in my elasticsearch.yml do? where do
>>> those logs appear?
>>>
>>> Thank you.
>>>
>>> On Wednesday, March 4, 2015 at 3:11:26 AM UTC+5:30, Mark Walkom wrote:

 You can set JAVA_OPTS in /etc/default/elasticsearch or
 /etc/sysconfig/elasticsearch u9nder ES_JAVA_OPTS

 On 3 March 2015 at 23:11, Anil Karaka  wrote:

> Some of the nodes are leaving the cluster from time to time..
>
> I un commented the GC Logging options in elasticsearch.yml file as
> shown and restarted my node.
>
> ## GC Logging
> 
>
> monitor.jvm.gc.young.warn: 1000ms
> monitor.jvm.gc.young.info: 700ms
> monitor.jvm.gc.young.debug: 400ms
>
> monitor.jvm.gc.old.warn: 10s
> monitor.jvm.gc.old.info: 5s
> monitor.jvm.gc.old.debug: 2s
>
>
> yet I'm not able to see any of the GC logging..
>
> I came across this blog post http://tech.taskrabbit.com/
> blog/2014/07/18/elasticsearch-in-production/
>
> `ElasticSearch will note in its normal logs if any GC is happening
> which effects a slow query, but we wanted more detail. Another logging
> option ElasticSearch has is ES_USE_GC_LOGGING which sets some JVM
> variables:
>
> # from elasticsearch.in.sh.erbif [ "x$ES_USE_GC_LOGGING" != "x" ]; then
>   JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails"
>   JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCTimeStamps"
>   JAVA_OPTS="$JAVA_OPTS -XX:+PrintClassHistogram"
>   JAVA_OPTS="$JAVA_OPTS -XX:+PrintTenuringDistribution"
>   JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCApplicationStoppedTime"
>   JAVA_OPTS="$JAVA_OPTS -Xloggc:/var/log/elasticsearch/gc.log"fi
>
> `
> Where do I set the ES_USE_GC_LOGGING option in my node..
> so that I can log GC events..
>
> currently there is no gc.log file in my logs folder..
> just normal cluster logs and slow index/search logs..
>
> Thanks..
>
>  --
> You received this message because you are subscribed to the Google
> Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to elasticsearc...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/elasticsearch/8cc75569-1b64-4883-b5d3-bd0f9e1b7312%
> 40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid

Re: Error Could not contact Elasticsearch at http://:9200. Please ensure that Elasticsearch is reachable from your system. 1 alert(s)

As per https://www.elastic.co/subscriptions/matrix we support most modern
browsers.
What do your webserver logs show?

On 1 April 2015 at 20:40, Sergey Zemlyanoy  wrote:

> ES - 1.4.2
> Kibana - 3.1.0
>
> Please note that it worked earlier for us
>
> Regards
> Sergey
>
> On Tuesday, March 31, 2015 at 5:37:20 AM UTC+3, Mark Walkom wrote:
>>
>> What versions of ES and KB are you running?
>>
>> On 30 March 2015 at 23:17, Sergey Zemlyanoy  wrote:
>>
>>> Hi,
>>>
>>> It's global problem in our environment. We cannot open Kibana UI all
>>> browsers but Internet Explorer.
>>> What might be misconfigured? ELK stack in general looks sane.
>>>
>>> Regards
>>> Sergey
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/elasticsearch/f0b39cd5-5ddf-42f9-b936-1dab13d0e0ac%
>>> 40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/312bef97-9378-49f2-b341-aa1d0b2b60f3%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-uWsTGrN31fszOruuK8opvyMAFgn_wj0oL-p4HmjziHw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: alias not indexing all documents

2015-04-01 Thread Stefanie

Hi,

I have created a gist with an example of what my alias and search filter 
look like.
https://gist.github.com/swarzech/4e2bcdcc046587fb2845

Thanks,
Stefanie

On Wednesday, April 1, 2015 at 5:15:25 PM UTC-4, Mark Walkom wrote:
>
> It'd help if you provided a gist/pastebin/etc link to the alias and the 
> filter.
>
> On 2 April 2015 at 02:51, Stefanie > 
> wrote:
>
>> Hi,
>>
>> I have an alias with filters that does not seem to be indexing any new 
>> documents.  When I run the filter directly on the index I get the correct 
>> results but when I call the alias the new documents are not being pulled.  
>>
>> This only seems to be happening with one alias as all of my other aliases 
>> that work the same way don't seem to be having a problem.  I have tried 
>> deleting the alias and recreating it but this didn't fix the issue.
>>
>> Does anyone know why this might be happening?
>>
>> Thank you,
>> Stefanie
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/48a5267d-7fa4-4d6e-bacb-54b93775955a%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/a7593519-ca8d-44cf-939a-bb688ec363e4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Restrict queries which are not fully defined

This is a little complex as you need to define what you think is a valid
query, and unfortunately there isn't a single solution here.

On 2 April 2015 at 03:59, Andy  wrote:

> To ES community,
> I have a large org to support where newbie users issue random/ adhoc
> queries across all indexes and fields slowing down the server.  While we
> are working on ramping up the storage etc -- i feel the need to restrict
> queries unless it is well defined.  A well defined query would be the one
> which specifies and index(1) and a field name (2).
>
> Requrement to restrict queries based on Index name (1) is captured in ER:
> https://github.com/elastic/elasticsearch/issues/6470
> For restricting queries based on field name (2), the way i understand, i
> can use "index.query.default_field" to point to insignificant column.
> This forces users to write correct queries for meaningful result.  Would
> this approach work?  Cons?
>
> Details:
> To try (2), i queried template on my instance but i do not see the
> parameter "index.query.default_field".  Below is the command i execute.
> curl -XGET 'localhost:9200/_template?pretty'
>
> To set the parameter, i need to do this:
>
> curl -XPUT http://localhost:9200/_template/ -d '
> {
> "template" : "logstash*",
> "settings" : {
> "index.query.default_field" : "@id"
>
>  }
> }
> '
>
> Would this prevent users through kibana to execute query and get meaningful 
> result unless they specify field name?  My understanding:
>
> Search string: * "Andy" *would return 0 results since it gets executed on 
> filed "id"
>
> Search string *name:"Andy" * would return correct results since it will 
> search only on field "name" across all indexes, if index is not specified.
>
> Is expectation correct?
>
> Appreciate quick comments from community.  Thanks!
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/5dda3a83-1df6-4aea-a37d-a59f17681af0%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8%3DenxP8DGenO9yd-kNPgCqKbCym2mfSNKL-Y6AEp9THw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: ES version and phonetic version

2015-04-01 Thread David Pilato

Is there a version 2.2.1?

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

> Le 1 avr. 2015 à 23:24, Mark Bramer  a écrit :
> 
> 
> Upon second review of the matrix, I realized that I AM in fact pairing a 
> supported version pair of ES 1.2.2 and phonetic 2.2.1.  So although my 
> version-matching questions are answered, I'm still stuck with why ES won't 
> start after plugin install.  I'm guessing that will require more details from 
> the log...
> 
> 
>> On Wednesday, April 1, 2015 at 5:20:42 PM UTC-4, Mark Bramer wrote:
>> 
>> I looked again at the plugin GitHub site and I now see the matrix (below) I 
>> should have seen before posting this.  My sincere apologies for not doing my 
>> homework, but I'm still interested in hearing any relevant details that may 
>> have popped into anyone's head while reading this.
>> 
>> elasticsearchPhonetic Analysis PluginDocs
>> master   Build from source   See below
>> es-1.x   Build from source   2.6.0-SNAPSHOT
>> es-1.5   2.5.0   2.5.0
>> es-1.4   2.4.2   2.4.2
>> < 1.4.3  2.4.1   2.4.1
>> es-1.3   2.3.0   2.3.0
>> es-1.2   2.2.0   2.2.0
>> es-1.1   2.1.0   2.1.0
>> es-1.0   2.0.0   2.0.0
>> es-0.90  1.8.0   1.8.0
>> 
>>> On Wednesday, April 1, 2015 at 5:17:42 PM UTC-4, Mark Bramer wrote:
>>> Hello,
>>> 
>>> In general, how important is it that plugin versions match the 
>>> Elasticsearch version?  Or are (can) plugins be versioned with no 
>>> correlation to Elasticsearch's version at all?  Specific case:
>>> 
>>> I have ES 1.2.2 and phonetic plugin 2.2.1.  In good practice, should I be 
>>> using ES 2.2.1?  Or vice versa; should I use phonetic 1.2.2?  so that the 
>>> versions match?
>>> 
>>> I was able to install the plugin on a 3-machine cluster using "plugin --url 
>>> file:///home/username/elasticsearch-analysis-phonetic-2.2.1 --install 
>>> phonetic-plugin" on all 3 machines (this may not be EXACTLY correct 
>>> command, I'm going by memory).  On all 3 machines, this created the 
>>> 'plugins' directory in my ES home and appeared to properly place the plugin 
>>> in the directory. I stopped ES using "service elasticsearch stop".  I tried 
>>> restarting using "... start" and ES will no longer start.  I see errors in 
>>> the logs that seem to pertain to plugins.  Given this, plus the fact ES 
>>> worked fine before the plugin installation, i uninstall the plugin using 
>>> "plugin --remove phonetic-plugin".  This removed the plugin from the 
>>> plugins directory but ES still won't start. The logs still contain 
>>> plugin-related errors.  Upon looking, I see the plugins directory is still 
>>> present.  I manually delete the directory, and now ES starts perfectly fine.
>>> 
>>> My main question is: Should my first course of troubleshooting be obtaining 
>>> a 1.2.2 version of the phonetic plugin?  Or is something else likely the 
>>> culprit?
>>> 
>>> *Some notes: 
>>> 1. This is a disconnected environment with no internet access, so I will 
>>> always have to do 'manual' installs.  
>>> 2. I currently do not have the exact errors to include here, but I can 
>>> hand-write some error details tomorrow when I'm back on the job if you want 
>>> more info.
>>> 
>>> Thanks, 
>>> Mark
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/879ec193-e550-48cc-af6d-9944c0904087%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/F300BB74-251D-46E8-BB42-0AF87EB538EC%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Re: alias not indexing all documents

2015-04-01 Thread Stefanie Warzecha

Hi,

I have created a gist with an example of what my alias and search filter
look like.
https://gist.github.com/swarzech/4e2bcdcc046587fb2845

Thanks,
Stefanie




*stefanie* warzechaco-founder & COOhttp://backstit.ch
 | @backstitch


On Wed, Apr 1, 2015 at 5:14 PM, Mark Walkom  wrote:

> It'd help if you provided a gist/pastebin/etc link to the alias and the
> filter.
>
> On 2 April 2015 at 02:51, Stefanie  wrote:
>
>> Hi,
>>
>> I have an alias with filters that does not seem to be indexing any new
>> documents.  When I run the filter directly on the index I get the correct
>> results but when I call the alias the new documents are not being pulled.
>>
>> This only seems to be happening with one alias as all of my other aliases
>> that work the same way don't seem to be having a problem.  I have tried
>> deleting the alias and recreating it but this didn't fix the issue.
>>
>> Does anyone know why this might be happening?
>>
>> Thank you,
>> Stefanie
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/48a5267d-7fa4-4d6e-bacb-54b93775955a%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/5gOGf1RMS4s/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9CG%3DiiBbLe-1i3G3OSHXP%2BbS6o32sz8LNNd5bGGdW4dA%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CANgshirWLa-K3ZdngmW1bqNAdJNgQKBwQcmuH0vxOy6KFnVA4Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: ES version and phonetic version


Upon second review of the matrix, I realized that I AM in fact pairing a 
supported version pair of ES 1.2.2 and phonetic 2.2.1.  So although my 
version-matching questions are answered, I'm still stuck with why ES won't 
start after plugin install.  I'm guessing that will require more details 
from the log...


On Wednesday, April 1, 2015 at 5:20:42 PM UTC-4, Mark Bramer wrote:
>
>
> I looked again at the plugin GitHub site and I now see the matrix (below) 
> I should have seen before posting this.  My sincere apologies for not doing 
> my homework, but I'm still interested in hearing any relevant details that 
> may have popped into anyone's head while reading this.
>
> elasticsearchPhonetic Analysis PluginDocsmasterBuild from sourceSee below
> es-1.xBuild from source2.6.0-SNAPSHOT 
> 
> es-1.52.5.02.5.0 
> 
> es-1.42.4.22.4.2 
> <
>  
> 1.4.32.4.12.4.1 
> 
> es-1.32.3.02.3.0 
> 
> es-1.22.2.02.2.0 
> 
> es-1.12.1.02.1.0 
> 
> es-1.02.0.02.0.0 
> 
> es-0.901.8.01.8.0 
> 
> On Wednesday, April 1, 2015 at 5:17:42 PM UTC-4, Mark Bramer wrote:
>>
>> Hello,
>>
>> In general, how important is it that plugin versions match the 
>> Elasticsearch version?  Or are (can) plugins be versioned with no 
>> correlation to Elasticsearch's version at all?  Specific case:
>>
>> I have ES 1.2.2 and phonetic plugin 2.2.1.  In good practice, should I be 
>> using ES 2.2.1?  Or vice versa; should I use phonetic 1.2.2?  so that the 
>> versions match?
>>
>> I was able to install the plugin on a 3-machine cluster using "plugin 
>> --url file:///home/username/elasticsearch-analysis-phonetic-2.2.1 --install 
>> phonetic-plugin" on all 3 machines (this may not be EXACTLY correct 
>> command, I'm going by memory).  On all 3 machines, this created the '
>> plugins' directory in my ES home and appeared to properly place the 
>> plugin in the directory. I stopped ES using "service elasticsearch stop". 
>>  I tried restarting using "... start" and ES will no longer start.  I 
>> see errors in the logs that seem to pertain to plugins.  Given this, plus 
>> the fact ES worked fine before the plugin installation, i uninstall the 
>> plugin using "plugin --remove phonetic-plugin".  This removed the plugin 
>> from the plugins directory but ES still won't start. The logs still contain 
>> plugin-related errors.  Upon looking, I see the plugins directory is still 
>> present.  I manually delete the directory, and now ES starts perfectly fine.
>>
>> My main question is: Should my first course of troubleshooting be 
>> obtaining a 1.2.2 version of the phonetic plugin?  Or is something else 
>> likely the culprit?
>>
>> *Some notes: 
>> 1. This is a disconnected environment with no internet access, so I will 
>> always have to do 'manual' installs.  
>> 2. I currently do not have the exact errors to include here, but I can 
>> hand-write some error details tomorrow when I'm back on the job if you want 
>> more info.
>>
>> Thanks, 
>> Mark
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/879ec193-e550-48cc-af6d-9944c0904087%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Auto Backup Cluster over JDBC MYSQL

You need to run the backup again, you can use cron to schedule this.

On 1 April 2015 at 20:02, Abdul Rafay  wrote:

> I have create a backup directory using FS Technique.
>
>
>
>  PUT _snapshot/databackup {
>"type": "fs",
>"settings": {
>   "location": "/mount/backups/my_backup"
>}}
>
> Then:
>
>  PUT _snapshot/my_backup/snapshot_2 {
>"indices": "completedata" }
>
>
> Its working.But when my database table update or insert new records backup 
> not update automatically.How Can i Do this auto
> backup after 1 day or by defining time.
>
> Thanks in advance.
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/8e7de3a1-0fb0-44bd-9f6f-08a8bc7e2101%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-w5-vYQi8_i6QaBu_RS31%3D9zhgiPFbqphvJVwOnX5Miw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Some of elasticsearch.yml configuration is not taking effect

(Use 
curl -XGET localhost:9200/_nodes
, they are not cluster level settings.

On 2 April 2015 at 00:34, Jakub Liska  wrote:

> Hey,
>
> I've asked this question on SO
> ,
> I have a problem, that settings from elasticsearch.yml like:
>
>  indices.memory.index_buffer_size: 50%
>
> Are not reflected in GET _cluster/settings as they should, right? Even
> though others like cluster-name or discovery are...
>
> What am I doing wrong here? I tried to restart ES with multiple ways of
> setting it up, nothing helps. Thank you
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/a0be7f56-da32-4b08-b304-3e99f2f3c9cf%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9%2BNxO2UKx8%2BsLxoAE0A4tjc3wBMTkK4gw9DHjUv5EqBQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: mlockall

The issue there is that these settings are node level, not cluster level.
As you mentioned, use curl -XGET localhost:9200/_nodes/


On 2 April 2015 at 01:21, Jakub Liska  wrote:

> I have the same problem with not only mlockall
> 
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/1f990d34-9ec9-41e3-8d79-e6bb53ac0428%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9Opnx_F%3DsLRVzZY496odTexth0Gq_XJEZLfW89qiWcbw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: ES version and phonetic version


I looked again at the plugin GitHub site and I now see the matrix (below) I 
should have seen before posting this.  My sincere apologies for not doing 
my homework, but I'm still interested in hearing any relevant details that 
may have popped into anyone's head while reading this.

elasticsearchPhonetic Analysis PluginDocsmasterBuild from sourceSee below
es-1.xBuild from source2.6.0-SNAPSHOT 

es-1.52.5.02.5.0 

es-1.42.4.22.4.2 
<
 
1.4.32.4.12.4.1 

es-1.32.3.02.3.0 

es-1.22.2.02.2.0 

es-1.12.1.02.1.0 

es-1.02.0.02.0.0 

es-0.901.8.01.8.0 

On Wednesday, April 1, 2015 at 5:17:42 PM UTC-4, Mark Bramer wrote:
>
> Hello,
>
> In general, how important is it that plugin versions match the 
> Elasticsearch version?  Or are (can) plugins be versioned with no 
> correlation to Elasticsearch's version at all?  Specific case:
>
> I have ES 1.2.2 and phonetic plugin 2.2.1.  In good practice, should I be 
> using ES 2.2.1?  Or vice versa; should I use phonetic 1.2.2?  so that the 
> versions match?
>
> I was able to install the plugin on a 3-machine cluster using "plugin 
> --url file:///home/username/elasticsearch-analysis-phonetic-2.2.1 --install 
> phonetic-plugin" on all 3 machines (this may not be EXACTLY correct 
> command, I'm going by memory).  On all 3 machines, this created the '
> plugins' directory in my ES home and appeared to properly place the 
> plugin in the directory. I stopped ES using "service elasticsearch stop". 
>  I tried restarting using "... start" and ES will no longer start.  I see 
> errors in the logs that seem to pertain to plugins.  Given this, plus the 
> fact ES worked fine before the plugin installation, i uninstall the plugin 
> using "plugin --remove phonetic-plugin".  This removed the plugin from 
> the plugins directory but ES still won't start. The logs still contain 
> plugin-related errors.  Upon looking, I see the plugins directory is still 
> present.  I manually delete the directory, and now ES starts perfectly fine.
>
> My main question is: Should my first course of troubleshooting be 
> obtaining a 1.2.2 version of the phonetic plugin?  Or is something else 
> likely the culprit?
>
> *Some notes: 
> 1. This is a disconnected environment with no internet access, so I will 
> always have to do 'manual' installs.  
> 2. I currently do not have the exact errors to include here, but I can 
> hand-write some error details tomorrow when I'm back on the job if you want 
> more info.
>
> Thanks, 
> Mark
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/ca9dd754-2aa1-472f-a6ee-ee91c23eabe3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

ES version and phonetic version

Hello,

In general, how important is it that plugin versions match the
Elasticsearch version? Or are (can) plugins be versioned with no
correlation to Elasticsearch's version at all? Specific case:

I have ES 1.2.2 and phonetic plugin 2.2.1. In good practice, should I be
using ES 2.2.1? Or vice versa; should I use phonetic 1.2.2? so that the
versions match?

I was able to install the plugin on a 3-machine cluster using "plugin --url
file:///home/username/elasticsearch-analysis-phonetic-2.2.1 --install
phonetic-plugin" on all 3 machines (this may not be EXACTLY correct
command, I'm going by memory). On all 3 machines, this created the 'plugins'
directory in my ES home and appeared to properly place the plugin in the
directory. I stopped ES using "service elasticsearch stop". I tried
restarting using "... start" and ES will no longer start. I see errors in
the logs that seem to pertain to plugins. Given this, plus the fact ES
worked fine before the plugin installation, i uninstall the plugin using
"plugin
--remove phonetic-plugin". This removed the plugin from the plugins
directory but ES still won't start. The logs still contain plugin-related
errors. Upon looking, I see the plugins directory is still present. I
manually delete the directory, and now ES starts perfectly fine.

My main question is: Should my first course of troubleshooting be obtaining
a 1.2.2 version of the phonetic plugin? Or is something else likely the
culprit?

*Some notes:
1. This is a disconnected environment with no internet access, so I will
always have to do 'manual' installs.
2. I currently do not have the exact errors to include here, but I can
hand-write some error details tomorrow when I'm back on the job if you want
more info.

Thanks,
Mark

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5e813536-7736-4f5f-ae55-d845c6662a7a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: mlockall

it's boostrap.mlcokall: true, not an equals, perhaps that is your problem?

On 1 April 2015 at 21:39, Cheten Dev  wrote:

> Hi,
>
> I don't know  what config sections you are asking about, but here you can
> see information about  two nodes
>
> i have uncommented bootstrap:mlockall = true
> i have tried ulimit -l unlimited
> i also tried the following ./bin/elasticsearch -Djna.tmpdir=/some path/
>
> but no success
>
>
> root@ubuntu:~/chetan/elasticsearch-1.5.0/bin# curl
> localhost:9200/_nodes/process ?pretty
> {
>   "cluster_name" : "CWESUBUNTU",
>   "nodes" : {
> "rD-BkWRnQ-uE5_VoWqStiA" : {
>   "name" : "ubuntu200",
>   "transport_address" : "inet[/192.168.1.200:9300]",
>   "host" : "ubuntu",
>   "ip" : "192.168.1.200",
>   "version" : "1.5.0",
>   "build" : "5448160",
>   "http_address" : "inet[/192.168.1.200:9200]",
>   "process" : {
> "refresh_interval_in_millis" : 1000,
> "id" : 12999,
> "max_file_descriptors" : 9,
> "mlockall" : false
>   }
> },
> "vGHvM7ncQ9id36ZcpSIFDQ" : {
>   "name" : "ubuntu202",
>   "transport_address" : "inet[/192.168.1.202:9300]",
>   "host" : "ubuntu02",
>   "ip" : "192.168.1.202",
>   "version" : "1.5.0",
>   "build" : "5448160",
>   "http_address" : "inet[/192.168.1.202:9200]",
>   "process" : {
> "refresh_interval_in_millis" : 1000,
> "id" : 12234,
> "max_file_descriptors" : 9,
> "mlockall" : true
>   }
> }
>   }
> }
>
>
> On Wed, Apr 1, 2015 at 5:28 AM, Mark Walkom  wrote:
>
>> Can you show us the relevant config sections from the nodes that don't
>> have mlockall set?
>>
>> On 31 March 2015 at 17:40, Cheten Dev  wrote:
>>
>>> Hi,
>>>
>>> here is the info
>>>
>>> Distributor ID: Ubuntu
>>> Description:Ubuntu 14.04.1 LTS
>>> Release:14.04
>>> Codename:   trusty
>>>
>>> I have tried to set mlockall = true on two server . i am able to
>>> successfully do it on one
>>> but not able to do it on other server
>>>
>>>
>>> On Tue, Mar 31, 2015 at 11:56 AM, Mark Walkom 
>>> wrote:
>>>
 Which linux?

 On 31 March 2015 at 16:16, Cheten Dev  wrote:

> Hi,
>
> I am on linux server with elasticsearch 1.5.0
>
> On Tue, Mar 31, 2015 at 8:07 AM, Mark Walkom 
> wrote:
>
>> A bit more info would be useful.
>>
>> What version of ES, what OS?
>>
>> On 30 March 2015 at 22:24, Chetan Dev  wrote:
>>
>>> Hi,
>>>
>>> i have tried everything written on the document to set mlackall =
>>> true  but
>>> nothing seems working here
>>>
>>> what am i missing here ?
>>>
>>>
>>> Thanks
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it,
>>> send an email to elasticsearch+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elasticsearch/5748d9fb-120a-4521-8caa-62b780ea0594%40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
>> You received this message because you are subscribed to a topic in
>> the Google Groups "elasticsearch" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/elasticsearch/R41hW2QaL0w/unsubscribe
>> .
>> To unsubscribe from this group and all its topics, send an email to
>> elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-92m9Qw4j4T8gwUV6%2BumkvbAOENTTsiupMurpGf8NZSw%40mail.gmail.com
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to the Google
> Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAG_NmL9T4dZeHUYpN0_5gwbfppL7%2BOJyNc6t3SQUpSe9N5oN2Q%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

  --
 You received this message because you are subscribed to a topic in the
 Google Groups

Re: alias not indexing all documents

It'd help if you provided a gist/pastebin/etc link to the alias and the
filter.

On 2 April 2015 at 02:51, Stefanie  wrote:

> Hi,
>
> I have an alias with filters that does not seem to be indexing any new
> documents.  When I run the filter directly on the index I get the correct
> results but when I call the alias the new documents are not being pulled.
>
> This only seems to be happening with one alias as all of my other aliases
> that work the same way don't seem to be having a problem.  I have tried
> deleting the alias and recreating it but this didn't fix the issue.
>
> Does anyone know why this might be happening?
>
> Thank you,
> Stefanie
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/48a5267d-7fa4-4d6e-bacb-54b93775955a%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9CG%3DiiBbLe-1i3G3OSHXP%2BbS6o32sz8LNNd5bGGdW4dA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Hadoop] Specifying username/password information for Shield-configured Elasticsearch

current = 2.0
2.1.Beta = 2.1 Beta
master = current dev version

On 4/1/15 11:59 PM, Michael Young wrote:

Ok, I found the docs with the referenced settings here:
http://www.elastic.co/guide/en/elasticsearch/hadoop/2.1.Beta/configuration.html

--
Michael

On Wed, Apr 1, 2015 at 4:54 PM, Michael Young mailto:michael.l.yo...@gmail.com>> wrote:

When I look at: 
http://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html 
there are no
references to es.net.http.auth.*

Am I missing it somewhere?  There are references to es.net.proxy.*, but not 
es.net.http.* that I can see.

Thank you again!

--
Michael

--
Michael

On Wed, Apr 1, 2015 at 4:44 PM, Costin Leau mailto:costin.l...@gmail.com>> wrote:

Actually they are also part of the es-hadoop documentation under the 
configuration section [1].
However when it comes to Shield, I find it easier to point to the 
relevant documentation section.

Cheers,

[1] 
http://www.elastic.co/guide/__en/elasticsearch/hadoop/__master/configuration.html#___basic_authentication

On 4/1/15 11:16 PM, Michael Young wrote:

Thank you!  I was looking in the ES-Hadoop documentation and didn't 
see these settings.  I didn't realize
they were in
the Shield documentation.  I'll give this a shot.

--
Michael

On Wed, Apr 1, 2015 at 2:44 PM, Costin Leau mailto:costin.l...@gmail.com>
>__> 
wrote:

 See this section of the docs:
http://www.elastic.co/guide/en/shield/current/hadoop.html

>

 On 4/1/15 9:35 PM, Michael Young wrote:

 I have Elasticsearch 1.4.4 and Shield 1.0.2 configured in 
my environment.  I'm able to successfully
connect to my
 cluster without issues using Active Directory as my 
authentication back-end.  I'm trying to use
es-hadoop to
 push data
 to Elasticsearch using Hive and/ Pig.  However, there 
doesn't seem to be any configuration settings
to specify a
 username/password for connecting to my Elasticsearch hosts 
for Hive or Pig EsStorageHandler.  Is
this an
 oversight?  Or
 am I missing something?

 --
 You received this message because you are subscribed to the Google 
Groups "elasticsearch" group.
 To unsubscribe from this group and stop receiving emails 
from it, send an email to
 elasticsearch+unsubscribe@__go__oglegroups.com 

__>

__>>.
 To view this discussion on the web visit

https://groups.google.com/d/msgid/elasticsearch/7e378659-225a-4895-b57a-1be2ab0fb98e%40googlegroups.com

>

>>.
 For more options, visit 
https://groups.google.com/d/optout

>.

Re: Tribe nodes

Please start your own thread for your question :)

On 2 April 2015 at 06:51, Deve java  wrote:

> hi members
>
> please, how we can use tribe node in java api ?
>
>
>
>
> Le jeudi 16 janvier 2014 17:52:25 UTC-5, Jörg Prante a écrit :
>>
>> 0.90.* and 1.0.0.RC1 node clients can not talk to each other, discovery
>> fails with errors like
>>
>> [2014-01-16 23:45:40,885][WARN ][discovery.zen.ping.multicast] [John
>> Falsworth] failed to read requesting data from /192.168.1.113:54328
>> java.io.IOException: No transport address mapped to [17519]
>> at org.elasticsearch.common.transport.TransportAddressSerializers.
>> addressFromStream(TransportAddressSerializers.java:71)
>>  at org.elasticsearch.cluster.node.DiscoveryNode.readFrom(
>> DiscoveryNode.java:267)
>> at org.elasticsearch.cluster.node.DiscoveryNode.readNode(
>> DiscoveryNode.java:257)
>>  at org.elasticsearch.discovery.zen.ping.multicast.
>> MulticastZenPing$Receiver.run(MulticastZenPing.java:410)
>> at java.lang.Thread.run(Thread.java:724)
>>
>> and tribe mode will not magically overcome this situation.
>>
>> Jörg
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/da91e510-81e3-4bd7-b01e-e2828ad2f0f6%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_uxzu%2Bhcws9KnrP96n0%2BLqN7xaDMYoYDb-NsXBjLxFRw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Hadoop] Specifying username/password information for Shield-configured Elasticsearch

Ok, I found the docs with the referenced settings here:
http://www.elastic.co/guide/en/elasticsearch/hadoop/2.1.Beta/configuration.html


--
Michael

On Wed, Apr 1, 2015 at 4:54 PM, Michael Young 
wrote:

> When I look at:
> http://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html
> there are no references to es.net.http.auth.*
>
> Am I missing it somewhere?  There are references to es.net.proxy.*, but
> not es.net.http.* that I can see.
>
> Thank you again!
>
> --
> Michael
>
> --
> Michael
>
> On Wed, Apr 1, 2015 at 4:44 PM, Costin Leau  wrote:
>
>> Actually they are also part of the es-hadoop documentation under the
>> configuration section [1].
>> However when it comes to Shield, I find it easier to point to the
>> relevant documentation section.
>>
>> Cheers,
>>
>> [1] http://www.elastic.co/guide/en/elasticsearch/hadoop/
>> master/configuration.html#_basic_authentication
>>
>> On 4/1/15 11:16 PM, Michael Young wrote:
>>
>>> Thank you!  I was looking in the ES-Hadoop documentation and didn't see
>>> these settings.  I didn't realize they were in
>>> the Shield documentation.  I'll give this a shot.
>>>
>>> --
>>> Michael
>>>
>>> On Wed, Apr 1, 2015 at 2:44 PM, Costin Leau >> > wrote:
>>>
>>> See this section of the docs:
>>> http://www.elastic.co/guide/__en/shield/current/hadoop.html <
>>> http://www.elastic.co/guide/en/shield/current/hadoop.html>
>>>
>>>
>>> On 4/1/15 9:35 PM, Michael Young wrote:
>>>
>>> I have Elasticsearch 1.4.4 and Shield 1.0.2 configured in my
>>> environment.  I'm able to successfully connect to my
>>> cluster without issues using Active Directory as my
>>> authentication back-end.  I'm trying to use es-hadoop to
>>> push data
>>> to Elasticsearch using Hive and/ Pig.  However, there doesn't
>>> seem to be any configuration settings to specify a
>>> username/password for connecting to my Elasticsearch hosts for
>>> Hive or Pig EsStorageHandler.  Is this an
>>> oversight?  Or
>>> am I missing something?
>>>
>>> --
>>> You received this message because you are subscribed to the
>>> Google Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from
>>> it, send an email to
>>> elasticsearch+unsubscribe@__googlegroups.com >> elasticsearch%2bunsubscr...@googlegroups.com>
>>> > elasticsearch%2bunsubscr...@googlegroups.com>>.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/__msgid/elasticsearch/7e378659-_
>>> _225a-4895-b57a-1be2ab0fb98e%__40googlegroups.com
>>> >> 225a-4895-b57a-1be2ab0fb98e%40googlegroups.com>
>>> >> __225a-4895-b57a-1be2ab0fb98e%__40googlegroups.com?utm_
>>> medium=__email&utm_source=footer
>>> >> 225a-4895-b57a-1be2ab0fb98e%40googlegroups.com?utm_medium=
>>> email&utm_source=footer>>.
>>> For more options, visit https://groups.google.com/d/__optout <
>>> https://groups.google.com/d/optout>.
>>>
>>>
>>> --
>>> Costin
>>>
>>> --
>>> You received this message because you are subscribed to a topic in
>>> the Google Groups "elasticsearch" group.
>>> To unsubscribe from this topic, visit https://groups.google.com/d/__
>>> topic/elasticsearch/__mlCw0uGov2s/unsubscribe
>>> >> mlCw0uGov2s/unsubscribe>.
>>> To unsubscribe from this group and all its topics, send an email to
>>> elasticsearch+unsubscribe@__googlegroups.com
>>> .
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/__msgid/elasticsearch/551C3CA6._
>>> _6090500%40gmail.com
>>> >> 6090500%40gmail.com>.
>>>
>>> For more options, visit https://groups.google.com/d/__optout <
>>> https://groups.google.com/d/optout>.
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to
>>> elasticsearch+unsubscr...@googlegroups.com >> unsubscr...@googlegroups.com>.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elasticsearch/CA%2B6%3DytWg-
>>> 6qFW4ErytAmjWQAhzyPrX1vxb9yCWSWgrLL_6D%2BXQ%40mail.gmail.com
>>> >> 6qFW4ErytAmjWQAhzyPrX1vxb9yCWSWgrLL_6D%2BXQ%40mail.gmail.
>>> com?utm_medium=email&utm_source=footer>.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
>> Costin
>>
>> --
>> You received this

Re: [Hadoop] Specifying username/password information for Shield-configured Elasticsearch

When I look at:
http://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html
there are no references to es.net.http.auth.*

Am I missing it somewhere?  There are references to es.net.proxy.*, but not
es.net.http.* that I can see.

Thank you again!

--
Michael

--
Michael

On Wed, Apr 1, 2015 at 4:44 PM, Costin Leau  wrote:

> Actually they are also part of the es-hadoop documentation under the
> configuration section [1].
> However when it comes to Shield, I find it easier to point to the relevant
> documentation section.
>
> Cheers,
>
> [1] http://www.elastic.co/guide/en/elasticsearch/hadoop/
> master/configuration.html#_basic_authentication
>
> On 4/1/15 11:16 PM, Michael Young wrote:
>
>> Thank you!  I was looking in the ES-Hadoop documentation and didn't see
>> these settings.  I didn't realize they were in
>> the Shield documentation.  I'll give this a shot.
>>
>> --
>> Michael
>>
>> On Wed, Apr 1, 2015 at 2:44 PM, Costin Leau > > wrote:
>>
>> See this section of the docs:
>> http://www.elastic.co/guide/__en/shield/current/hadoop.html <
>> http://www.elastic.co/guide/en/shield/current/hadoop.html>
>>
>>
>> On 4/1/15 9:35 PM, Michael Young wrote:
>>
>> I have Elasticsearch 1.4.4 and Shield 1.0.2 configured in my
>> environment.  I'm able to successfully connect to my
>> cluster without issues using Active Directory as my
>> authentication back-end.  I'm trying to use es-hadoop to
>> push data
>> to Elasticsearch using Hive and/ Pig.  However, there doesn't
>> seem to be any configuration settings to specify a
>> username/password for connecting to my Elasticsearch hosts for
>> Hive or Pig EsStorageHandler.  Is this an
>> oversight?  Or
>> am I missing something?
>>
>> --
>> You received this message because you are subscribed to the
>> Google Groups "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to
>> elasticsearch+unsubscribe@__googlegroups.com > elasticsearch%2bunsubscr...@googlegroups.com>
>>  elasticsearch%2bunsubscr...@googlegroups.com>>.
>> To view this discussion on the web visit
>> https://groups.google.com/d/__msgid/elasticsearch/7e378659-_
>> _225a-4895-b57a-1be2ab0fb98e%__40googlegroups.com
>> > 225a-4895-b57a-1be2ab0fb98e%40googlegroups.com>
>> > __225a-4895-b57a-1be2ab0fb98e%__40googlegroups.com?utm_
>> medium=__email&utm_source=footer
>> > 225a-4895-b57a-1be2ab0fb98e%40googlegroups.com?utm_medium=
>> email&utm_source=footer>>.
>> For more options, visit https://groups.google.com/d/__optout <
>> https://groups.google.com/d/optout>.
>>
>>
>> --
>> Costin
>>
>> --
>> You received this message because you are subscribed to a topic in
>> the Google Groups "elasticsearch" group.
>> To unsubscribe from this topic, visit https://groups.google.com/d/__
>> topic/elasticsearch/__mlCw0uGov2s/unsubscribe
>> > mlCw0uGov2s/unsubscribe>.
>> To unsubscribe from this group and all its topics, send an email to
>> elasticsearch+unsubscribe@__googlegroups.com
>> .
>> To view this discussion on the web visit
>> https://groups.google.com/d/__msgid/elasticsearch/551C3CA6._
>> _6090500%40gmail.com
>> > 6090500%40gmail.com>.
>>
>> For more options, visit https://groups.google.com/d/__optout <
>> https://groups.google.com/d/optout>.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to
>> elasticsearch+unsubscr...@googlegroups.com > unsubscr...@googlegroups.com>.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CA%2B6%3DytWg-
>> 6qFW4ErytAmjWQAhzyPrX1vxb9yCWSWgrLL_6D%2BXQ%40mail.gmail.com
>> > 6qFW4ErytAmjWQAhzyPrX1vxb9yCWSWgrLL_6D%2BXQ%40mail.gmail.
>> com?utm_medium=email&utm_source=footer>.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> Costin
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/elasticsearch/mlCw0uGov2s/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this di

Re: [Hadoop] Specifying username/password information for Shield-configured Elasticsearch

Actually they are also part of the es-hadoop documentation under the
configuration section [1].
However when it comes to Shield, I find it easier to point to the relevant
documentation section.

Cheers,

[1]
http://www.elastic.co/guide/en/elasticsearch/hadoop/master/configuration.html#_basic_authentication

On 4/1/15 11:16 PM, Michael Young wrote:

Thank you! I was looking in the ES-Hadoop documentation and didn't see these
settings. I didn't realize they were in
the Shield documentation. I'll give this a shot.

--
Michael

On Wed, Apr 1, 2015 at 2:44 PM, Costin Leau mailto:costin.l...@gmail.com>> wrote:

See this section of the docs:
http://www.elastic.co/guide/__en/shield/current/hadoop.html

On 4/1/15 9:35 PM, Michael Young wrote:

I have Elasticsearch 1.4.4 and Shield 1.0.2 configured in my
environment. I'm able to successfully connect to my
cluster without issues using Active Directory as my authentication
back-end. I'm trying to use es-hadoop to
push data
to Elasticsearch using Hive and/ Pig. However, there doesn't seem to
be any configuration settings to specify a
username/password for connecting to my Elasticsearch hosts for Hive or
Pig EsStorageHandler. Is this an
oversight? Or
am I missing something?

>.
To view this discussion on the web visit

https://groups.google.com/d/__msgid/elasticsearch/7e378659-__225a-4895-b57a-1be2ab0fb98e%__40googlegroups.com

>.
For more options, visit https://groups.google.com/d/__optout
.

--
Costin

--
You received this message because you are subscribed to a topic in the Google Groups
"elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/__topic/elasticsearch/__mlCw0uGov2s/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@__googlegroups.com
.
To view this discussion on the web visit

https://groups.google.com/d/__msgid/elasticsearch/551C3CA6.__6090500%40gmail.com

For more options, visit https://groups.google.com/d/__optout
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
elasticsearch+unsubscr...@googlegroups.com
.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CA%2B6%3DytWg-6qFW4ErytAmjWQAhzyPrX1vxb9yCWSWgrLL_6D%2BXQ%40mail.gmail.com
.
For more options, visit https://groups.google.com/d/optout.

--
Costin

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/551C58AB.2010501%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Copying fields to a geopoint type ?

2015-04-01 Thread Alex Schokking

Woah crazy, never would've thought of that, thanks a lot for following up!

On Wed, Apr 1, 2015 at 12:31 PM, Pascal VINCENT 
wrote:

> I finally come up with :
>
>  if [latitude] and [longitude] {
> mutate {
> add_field => [ "[location]", "%{longitude}" ]
> add_field => [ "[location]", "%{latitude}" ]
> }
> mutate {
> convert => [ "[location]", "float" ]
> }
>   }
>
>
  --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/QaI1fj74RlM/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/47c8334d-109e-4052-9973-afa69dd49709%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>



-- 
-- Alex

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CACXbQESyNcYr9g1u2dEFRovHE-NtB_JwugNn76G7_36Tm0Mteg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Hadoop] Specifying username/password information for Shield-configured Elasticsearch

Thank you!  I was looking in the ES-Hadoop documentation and didn't see
these settings.  I didn't realize they were in the Shield documentation.
I'll give this a shot.

--
Michael

On Wed, Apr 1, 2015 at 2:44 PM, Costin Leau  wrote:

> See this section of the docs:
> http://www.elastic.co/guide/en/shield/current/hadoop.html
>
>
> On 4/1/15 9:35 PM, Michael Young wrote:
>
>> I have Elasticsearch 1.4.4 and Shield 1.0.2 configured in my
>> environment.  I'm able to successfully connect to my
>> cluster without issues using Active Directory as my authentication
>> back-end.  I'm trying to use es-hadoop to push data
>> to Elasticsearch using Hive and/ Pig.  However, there doesn't seem to be
>> any configuration settings to specify a
>> username/password for connecting to my Elasticsearch hosts for Hive or
>> Pig EsStorageHandler.  Is this an oversight?  Or
>> am I missing something?
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to
>> elasticsearch+unsubscr...@googlegroups.com > unsubscr...@googlegroups.com>.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/7e378659-
>> 225a-4895-b57a-1be2ab0fb98e%40googlegroups.com
>> > 225a-4895-b57a-1be2ab0fb98e%40googlegroups.com?utm_medium=
>> email&utm_source=footer>.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> Costin
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/elasticsearch/mlCw0uGov2s/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/elasticsearch/551C3CA6.6090500%40gmail.com.
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CA%2B6%3DytWg-6qFW4ErytAmjWQAhzyPrX1vxb9yCWSWgrLL_6D%2BXQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Tribe nodes

2015-04-01 Thread Deve java

hi members 

please, how we can use tribe node in java api ?




Le jeudi 16 janvier 2014 17:52:25 UTC-5, Jörg Prante a écrit :
>
> 0.90.* and 1.0.0.RC1 node clients can not talk to each other, discovery 
> fails with errors like
>
> [2014-01-16 23:45:40,885][WARN ][discovery.zen.ping.multicast] [John 
> Falsworth] failed to read requesting data from /192.168.1.113:54328
> java.io.IOException: No transport address mapped to [17519]
> at 
> org.elasticsearch.common.transport.TransportAddressSerializers.addressFromStream(TransportAddressSerializers.java:71)
>  at 
> org.elasticsearch.cluster.node.DiscoveryNode.readFrom(DiscoveryNode.java:267)
> at 
> org.elasticsearch.cluster.node.DiscoveryNode.readNode(DiscoveryNode.java:257)
>  at 
> org.elasticsearch.discovery.zen.ping.multicast.MulticastZenPing$Receiver.run(MulticastZenPing.java:410)
> at java.lang.Thread.run(Thread.java:724)
>
> and tribe mode will not magically overcome this situation.
>
> Jörg
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/da91e510-81e3-4bd7-b01e-e2828ad2f0f6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Copying fields to a geopoint type ?

2015-04-01 Thread Pascal VINCENT

I finally come up with :

 if [latitude] and [longitude] {
mutate {
add_field => [ "[location]", "%{longitude}" ]
add_field => [ "[location]", "%{latitude}" ]
}
mutate {
convert => [ "[location]", "float" ]
}
  } 


>>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/47c8334d-109e-4052-9973-afa69dd49709%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Shard Allocation Problem

2015-04-01 Thread Marcelo Paes Rech

The ElasticSearch Team closed the issue, but I didn't have an answer. I 
still looking for a solution.


Regards.

Em sábado, 28 de março de 2015 16:29:05 UTC-3, Marcelo Paes Rech escreveu:
>
> Hi guys,
>
> Our cluster has 7 data nodes. Last week one of them was turned off. We 
> restarted the node and then the cluster started rebalancing. The problem is 
> it never finishes rebalancing.
>
> Node logs shows an update coming from master node during the rebalancing, 
> seconds later the rebalancing is canceled and then it starts again.
>
> I have created #10281 
>  on GitHub, but I 
> am including the problem here too. Maybe somebody ran into this problem and 
> already fixed it.
>
> The logs were collected in different moments but they represent the same 
> behaviour.
>
> The log follows:
>
> Data Node
>
> https://gist.github.com/marcelopaesrech/7b9f94310ff7a24bed9e
>
> Master Node
>
> https://gist.github.com/marcelopaesrech/c80542a85c83d9c1756d
>
>
> Regards.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d7e23198-fd2a-4b41-aeed-cf9b8cc23ae7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Hadoop] Specifying username/password information for Shield-configured Elasticsearch


See this section of the docs:
http://www.elastic.co/guide/en/shield/current/hadoop.html


On 4/1/15 9:35 PM, Michael Young wrote:

I have Elasticsearch 1.4.4 and Shield 1.0.2 configured in my environment.  I'm 
able to successfully connect to my
cluster without issues using Active Directory as my authentication back-end.  
I'm trying to use es-hadoop to push data
to Elasticsearch using Hive and/ Pig.  However, there doesn't seem to be any 
configuration settings to specify a
username/password for connecting to my Elasticsearch hosts for Hive or Pig 
EsStorageHandler.  Is this an oversight?  Or
am I missing something?

--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to
elasticsearch+unsubscr...@googlegroups.com 
.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/7e378659-225a-4895-b57a-1be2ab0fb98e%40googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.


--
Costin

--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/551C3CA6.6090500%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Hadoop] Specifying username/password information for Shield-configured Elasticsearch

I have Elasticsearch 1.4.4 and Shield 1.0.2 configured in my environment.  
I'm able to successfully connect to my cluster without issues using Active 
Directory as my authentication back-end.  I'm trying to use es-hadoop to 
push data to Elasticsearch using Hive and/ Pig.  However, there doesn't 
seem to be any configuration settings to specify a username/password for 
connecting to my Elasticsearch hosts for Hive or Pig EsStorageHandler.  Is 
this an oversight?  Or am I missing something?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/7e378659-225a-4895-b57a-1be2ab0fb98e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Problem with ES- Shield

2015-04-01 Thread 'BEN SALEM Omar' via elasticsearch

When I try to add an esuser and grant him a role:

bin/shield/esusers useradd rdeniro -p taxidriver -r admin

It says that there is an error in the "esusers" file in the
elef['$ES_INCLUDE"] line, I think the line 31 or 38, I forgot.
In the line above, there is a $...done: which ends with ":", if you change
it to done; with ";" , the error doesn't show up anymore.

I'm installing the whole thing in the Hortonworks Sandbox 2.0 vm
ᐧ

On Wed, Apr 1, 2015 at 8:14 PM, Jay Modi  wrote:

> I'm glad you were able to fix this issue. I have a few questions that may
> help us ensure this fix is pulled back into Shield. What OS and Shell are
> you using? Which line in the script did you change exactly? I'm looking at
> the script right now and I don't see any instance of "done:"
>
> Just an FYI, there is another way to handle the configuration directory
> issue that is documented in the configuring your environment section of the
> shield guide:
>
> http://www.elastic.co/guide/en/shield/current/getting-started.html#_configuring_your_environment
>
> On Wednesday, April 1, 2015 at 2:07:28 PM UTC-4, BEN SALEM Omar wrote:
>>
>> Ok, I've managed to solve my own problem.
>>
>> First of all things, if you want to just remove the shield plugin run
>> under the elasticsearch :
>>
>> $ bin/plugin --remove shield
>>
>> Now, to the syntax error in the elef line in the "esusers", generally,
>> when there is such an error, it's related to things before that line.
>> In fact there was something like this :
>> $done:
>> correction :
>> $done;
>>
>> Now, we could add users within shield and grunt them the desired role.
>>
>> But, even if I've done it, I couldn't access to my ES cluster with the
>> "new born user"; I think because my ES is running as a service and not
>> starting with bin/elasticsearch, that lead to "misunderstanding."
>>
>> The solution I found was to transfer the 
>> /usr/share/elasticsearch/config/shield
>> under /etc/elasticsearch
>> $ln -s /usr/share/elasticsearch/config/shield/ /etc/elasticsearch
>>
>> And that was it !
>> Hope that could be of a help for anyone else facing the same problem !
>>
>> On Wednesday, April 1, 2015 at 5:28:14 PM UTC+2, BEN SALEM Omar wrote:
>>>
>>> I wanted to add authentication to my ES, thus I downloaded the shield :
>>> https://www.elastic.co/downloads/shield
>>>
>>> Then I've tried to add a user to give him the admin role as it's
>>> mentionned here :
>>> http://www.elastic.co/guide/en/shield/current/getting-started.html
>>> (I've added it with root, cause I start my es with root..)
>>>
>>> But when I try to do so, an error occure saying that there is a syntax
>>> error in the 38 line "the elef line"
>>> elef[$ES_INCLUDE] bla bla
>>> I've tried many time :
>>> bin/shield/esusers useradd omar -p omar -r admin
>>> bin/shield/esusers useradd esomar -p omar -r admin
>>> bin/shield/esusers useradd es_omar -p omar -r admin
>>> bin/shield/esusers useradd ES_omar -p omar -r admin
>>> bin/shield/esusers useradd omar  -r admin
>>> bin/shield/esusers useradd esomar  -r admin
>>> bin/shield/esusers useradd es_omar  -r admin
>>> bin/shield/esusers useradd ES_omar  -r admin
>>>
>>> But the same error kept on jumping !
>>>
>>> I said enough is enough and wanted to keep on doing wwhat I was doing,
>>> tried to run Kibana, and this is what happened !
>>>
>>>
>>> I've then tried to open my ES marvel plugin, but there is now an
>>> authentication box
>>>
>>>
>>> 
>>> I've tried ALL the combinations above, but none of them seemed to work !!
>>>
>>> I'm really preoccupied by this note :
>>> To ensure that Elasticsearch can read the user and role information at
>>> startup, run esusers useradd as the same user you use to run
>>> Elasticsearch. Running the command as root or some other user will update
>>> the permissions for the users and users_roles files and prevent
>>> Elasticsearch from accessing them.
>>>
>>> The thing is that I start my es and kibana with the root..
>>> Please HELP !!
>>> ᐧ
>>>
>>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/sEZpCO7E_B0/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/9f0ebe63-cc83-4c0b-8ee9-1514e51f77f6%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msg

Re: Problem with ES- Shield

2015-04-01 Thread Jay Modi

I'm glad you were able to fix this issue. I have a few questions that may 
help us ensure this fix is pulled back into Shield. What OS and Shell are 
you using? Which line in the script did you change exactly? I'm looking at 
the script right now and I don't see any instance of "done:"

Just an FYI, there is another way to handle the configuration directory 
issue that is documented in the configuring your environment section of the 
shield guide:
http://www.elastic.co/guide/en/shield/current/getting-started.html#_configuring_your_environment

On Wednesday, April 1, 2015 at 2:07:28 PM UTC-4, BEN SALEM Omar wrote:
>
> Ok, I've managed to solve my own problem.
>
> First of all things, if you want to just remove the shield plugin run 
> under the elasticsearch :
>
> $ bin/plugin --remove shield
>
> Now, to the syntax error in the elef line in the "esusers", generally, 
> when there is such an error, it's related to things before that line.
> In fact there was something like this : 
> $done: 
> correction : 
> $done;
>
> Now, we could add users within shield and grunt them the desired role.
>
> But, even if I've done it, I couldn't access to my ES cluster with the 
> "new born user"; I think because my ES is running as a service and not 
> starting with bin/elasticsearch, that lead to "misunderstanding."
>
> The solution I found was to transfer the 
> /usr/share/elasticsearch/config/shield under /etc/elasticsearch
> $ln -s /usr/share/elasticsearch/config/shield/ /etc/elasticsearch
>
> And that was it ! 
> Hope that could be of a help for anyone else facing the same problem !
>
> On Wednesday, April 1, 2015 at 5:28:14 PM UTC+2, BEN SALEM Omar wrote:
>>
>> I wanted to add authentication to my ES, thus I downloaded the shield :
>> https://www.elastic.co/downloads/shield
>>
>> Then I've tried to add a user to give him the admin role as it's 
>> mentionned here :
>> http://www.elastic.co/guide/en/shield/current/getting-started.html
>> (I've added it with root, cause I start my es with root..)
>>
>> But when I try to do so, an error occure saying that there is a syntax 
>> error in the 38 line "the elef line" 
>> elef[$ES_INCLUDE] bla bla
>> I've tried many time : 
>> bin/shield/esusers useradd omar -p omar -r admin
>> bin/shield/esusers useradd esomar -p omar -r admin
>> bin/shield/esusers useradd es_omar -p omar -r admin
>> bin/shield/esusers useradd ES_omar -p omar -r admin
>> bin/shield/esusers useradd omar  -r admin
>> bin/shield/esusers useradd esomar  -r admin
>> bin/shield/esusers useradd es_omar  -r admin
>> bin/shield/esusers useradd ES_omar  -r admin
>>
>> But the same error kept on jumping ! 
>>
>> I said enough is enough and wanted to keep on doing wwhat I was doing, 
>> tried to run Kibana, and this is what happened !
>>
>>
>> I've then tried to open my ES marvel plugin, but there is now an 
>> authentication box 
>>
>>
>> 
>> I've tried ALL the combinations above, but none of them seemed to work !!
>>
>> I'm really preoccupied by this note : 
>> To ensure that Elasticsearch can read the user and role information at 
>> startup, run esusers useradd as the same user you use to run 
>> Elasticsearch. Running the command as root or some other user will update 
>> the permissions for the users and users_roles files and prevent 
>> Elasticsearch from accessing them.
>>
>> The thing is that I start my es and kibana with the root..
>> Please HELP !! 
>> ᐧ
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/9f0ebe63-cc83-4c0b-8ee9-1514e51f77f6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Language specific analyzer in the query

2015-04-01 Thread werewolf420

For example,

I have a field named text_fr (french analyzed), when I search with:

{
"query": {
"query_string": {
"query": "person",
"_name": "person",
"analyzer": "french"
}
}
}

I dont get any results. However if I search with

{
"query": {
"query_string": {
"query": "text_fr:person",
"_name": "text_fr:person"
}
}
}

Is there a away I can retrieve the documents with "personne" (documents 
retrieved in the second query) by using all (not field specific) query. 
Hope this adds more info on my use case.
On Wednesday, April 1, 2015 at 10:40:24 AM UTC-7, werewolf420 wrote:
>
> Hi,
>
> I have a document with different text fields. For example I have a text 
> field for french text (text_fr) which has a french analyzer set in the 
> mapping. Once the indexing is done and I try to query the; field specific 
> (text_fr:"xyz") querying works well with fuzziness (searching for word 
> "person" will return the documents with "personne"). However if I do a all 
> search with "person" it doesn't return anything. I tried applying french 
> analyzer for the query but no results.
>
> Any pointers to what I am missing will be of great help.
>
> Cheers 
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/60be99e5-575a-477d-ae6c-f699bd643bae%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: URGENT HELP : can't connect to my ES and Kibana after installing shield !

Ok, I've managed to solve my own problem.

First of all things, if you want to just remove the shield plugin run under 
the elasticsearch :

$ bin/plugin --remove shield

Now, to the syntax error in the elef line in the "esusers", generally, when 
there is such an error, it's related to things before that line.
In fact there was something like this : 
$done: 
correction : 
$done;

Now, we could add users within shield and grunt them the desired role.

But, even if I've done it, I couldn't access to my ES cluster with the "new 
born user"; I think because my ES is running as a service and not starting 
with bin/elasticsearch, that lead to "misunderstanding."

The solution I found was to transfer the 
/usr/share/elasticsearch/config/shield under /etc/elasticsearch
$ln -s /usr/share/elasticsearch/config/shield/ /etc/elasticsearch

And that was it ! 
Hope that could be of a help for anyone else facing the same problem !

On Wednesday, April 1, 2015 at 4:19:52 PM UTC+2, BEN SALEM Omar wrote:
>
> Hi everyone, 
>
> I've installed the shield to add authentication to my ES ! 
>
> When I've tried to add a user with an admin role, an error has occured 
> saying that there is a syntax error in the elif[$ES_INCLUDE]
>
> (I've created the user with the root since I start my ES with the root 
> user !)
>
> Now, when I try to run Kibana, this error jumps up : 
>
>
> 
> And when I try to access to my Marvel plugin, a box showing authentication 
> required jumps to my face ! when I enter the user I've created (with the 
> error I mentionned) , I can't access to it !!!
>
>
> What to do please??! I'm in a huge hurry !
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1e32b404-db19-4679-89e9-c608c9e67c30%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Problem with ES- Shield

Ok, I've managed to solve my own problem.

First of all things, if you want to just remove the shield plugin run under 
the elasticsearch :

$ bin/plugin --remove shield

Now, to the syntax error in the elef line in the "esusers", generally, when 
there is such an error, it's related to things before that line.
In fact there was something like this : 
$done: 
correction : 
$done;

Now, we could add users within shield and grunt them the desired role.

But, even if I've done it, I couldn't access to my ES cluster with the "new 
born user"; I think because my ES is running as a service and not starting 
with bin/elasticsearch, that lead to "misunderstanding."

The solution I found was to transfer the 
/usr/share/elasticsearch/config/shield under /etc/elasticsearch
$ln -s /usr/share/elasticsearch/config/shield/ /etc/elasticsearch

And that was it ! 
Hope that could be of a help for anyone else facing the same problem !

On Wednesday, April 1, 2015 at 5:28:14 PM UTC+2, BEN SALEM Omar wrote:
>
> I wanted to add authentication to my ES, thus I downloaded the shield :
> https://www.elastic.co/downloads/shield
>
> Then I've tried to add a user to give him the admin role as it's 
> mentionned here :
> http://www.elastic.co/guide/en/shield/current/getting-started.html
> (I've added it with root, cause I start my es with root..)
>
> But when I try to do so, an error occure saying that there is a syntax 
> error in the 38 line "the elef line" 
> elef[$ES_INCLUDE] bla bla
> I've tried many time : 
> bin/shield/esusers useradd omar -p omar -r admin
> bin/shield/esusers useradd esomar -p omar -r admin
> bin/shield/esusers useradd es_omar -p omar -r admin
> bin/shield/esusers useradd ES_omar -p omar -r admin
> bin/shield/esusers useradd omar  -r admin
> bin/shield/esusers useradd esomar  -r admin
> bin/shield/esusers useradd es_omar  -r admin
> bin/shield/esusers useradd ES_omar  -r admin
>
> But the same error kept on jumping ! 
>
> I said enough is enough and wanted to keep on doing wwhat I was doing, 
> tried to run Kibana, and this is what happened !
>
>
> I've then tried to open my ES marvel plugin, but there is now an 
> authentication box 
>
>
> 
> I've tried ALL the combinations above, but none of them seemed to work !!
>
> I'm really preoccupied by this note : 
> To ensure that Elasticsearch can read the user and role information at 
> startup, run esusers useradd as the same user you use to run 
> Elasticsearch. Running the command as root or some other user will update 
> the permissions for the users and users_roles files and prevent 
> Elasticsearch from accessing them.
>
> The thing is that I start my es and kibana with the root..
> Please HELP !! 
> ᐧ
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/8d217029-3548-409c-8501-540098292265%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch 1.4.4-1 with Shield 1.0.1 on CentOS 6.6 - Authentication issue when running as service vs bin/elasticsearch.

2015-04-01 Thread Jay Modi

If you set the proper configuration directory in the environment variable 
and run the esusers commands again, then your elasticsearch instance should 
be accessible.

If you must remove Shield: bin/plugin --remove shield
http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-plugins.html#_removing_plugins

On Wednesday, April 1, 2015 at 12:09:54 PM UTC-4, Lobna Tonn wrote:
>
>
> How to delete this shield plugin pleaase? I'm facing the same error since 
> I have my ES running as a service !
> On Monday, February 23, 2015 at 4:28:32 PM UTC+1, Jason wrote:
>>
>> I am able to run bin/elasticsearch and authenticate just fine but 
>> whenever I run elasticsearch as a service, I consistently receive the 
>> following error:
>>
>> {
>>   "error" : "AuthenticationException[unable to authenticate user 
>> [rdeniro] for REST request [/idx1?pretty]]",
>>   "status" : 401
>> }
>>
>> I made sure that the $ES_HOME/config/shield/ dir and files were all owned 
>> by ES user and tried to dig down to see how the two startup methods differ. 
>> Ultimately, it came down to something in 
>> /etc/elasticsearch/elasticsearch.yml by comparing the two runtime 
>> environments, but haven't been able to make much headway from there.
>>
>> Anyone have any thoughts on this? Thanks in advance for any help.
>>
>> Cheers,
>> Jason
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f37e4ae8-99c8-4b42-bb0a-6dd4faa4d2a9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Language specific analyzer in the query

2015-04-01 Thread werewolf420

Hi,

I have a document with different text fields. For example I have a text 
field for french text (text_fr) which has a french analyzer set in the 
mapping. Once the indexing is done and I try to query the; field specific 
(text_fr:"xyz") querying works well with fuzziness (searching for word 
"person" will return the documents with "personne"). However if I do a all 
search with "person" it doesn't return anything. I tried applying french 
analyzer for the query but no results.

Any pointers to what I am missing will be of great help.

Cheers 

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4d38596f-6ceb-45ec-923c-885b82320f06%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Restrict queries which are not fully defined

2015-04-01 Thread Andy

To ES community,
I have a large org to support where newbie users issue random/ adhoc
queries across all indexes and fields slowing down the server. While we
are working on ramping up the storage etc -- i feel the need to restrict
queries unless it is well defined. A well defined query would be the one
which specifies and index(1) and a field name (2).

Requrement to restrict queries based on Index name (1) is captured in ER:
https://github.com/elastic/elasticsearch/issues/6470
For restricting queries based on field name (2), the way i understand, i
can use "index.query.default_field" to point to insignificant column. This
forces users to write correct queries for meaningful result. Would this
approach work? Cons?

Details:
To try (2), i queried template on my instance but i do not see the
parameter "index.query.default_field". Below is the command i execute.
curl -XGET 'localhost:9200/_template?pretty'

To set the parameter, i need to do this:

curl -XPUT http://localhost:9200/_template/ -d '
{
"template" : "logstash*",
"settings" : {
"index.query.default_field" : "@id"

}
}
'

Would this prevent users through kibana to execute query and get meaningful
result unless they specify field name? My understanding:

Search string: * "Andy" *would return 0 results since it gets executed on filed
"id"

Search string *name:"Andy" * would return correct results since it will search
only on field "name" across all indexes, if index is not specified.

Is expectation correct?

Appreciate quick comments from community. Thanks!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5dda3a83-1df6-4aea-a37d-a59f17681af0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to get Elasticsearch running on port 9200?

2015-04-01 Thread Hichem MAALMI

Try closing any other program (including ES instance) using the same port 
before running it, otherwise i think elasticsearch will try the next 
available port 9201 9202 etc ..

On Monday, March 30, 2015 at 3:45:34 PM UTC+2, kelnrluierhfeulne wrote:
>
> What do you do to get Elasticsearch running on port 9200?
> Thanks!!
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/46419fc6-1759-4710-9365-3399b42d3712%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Copying fields to a geopoint type ?

2015-04-01 Thread Alex Schokking

Were you ever able to figure out a solution to this? I'm in a similar boat.

On Thursday, September 11, 2014 at 2:14:29 AM UTC-7, Kushal Zamkade wrote:
>
> Hello,
>
> I have created a location filed by using below code 
>
>  if [latitude] and [longitude] {
>
> mutate {
>   rename => [ "latitude", "[location][lat]", "longitude", 
> "[location][lon]" ]
>
> }
>   }
>
> But when i check location field type then it is not created as geo_point.
>
> when i am trying to search a geo_point then i am getting below error.
>  QueryParsingException[[logstash-2014.09.11] failed to find geo_point field 
> [location1]]; 
>
> can you help me to resolve this
>
>
>
> On Thursday, April 10, 2014 2:42:22 AM UTC+5:30, Pascal VINCENT wrote:
>>
>> Hi,
>>
>> I have included logstash in my stack and started to play with it. I'm 
>> sure it can do the trick I was looking for, and much more.
>> Thank you ... 
>>
>> [waiting for your blog post :)] 
>>
>> Pascal. 
>>
>>
>> On Mon, Apr 7, 2014 at 9:38 AM, Alexander Reelsen  
>> wrote:
>>
>>> Hey,
>>>
>>> I dont know about your stack, but maybe logstash would be a good idea to 
>>> add it in there. It is more flexible than the csv river and features a CSV 
>>> input as well. You can easily change the structure of the data you want to 
>>> index. This is how the logstash config would look like
>>>
>>>   if [latitude] and [longitude] {
>>>
>>> mutate {
>>>   rename => [ "latitude", "[location][lat]", "longitude", 
>>> "[location][lon]" ]
>>>
>>> }
>>>   }
>>>
>>> I am currently working on a blog post how to utilize elasticsearch, 
>>> logstash and kibana on CSV based data and hope to release it soonish on the 
>>> .org blog - which covers exactly this. Stay tuned! :-)
>>>
>>>
>>> --Alex
>>>
>>>
>>>
>>> On Thu, Apr 3, 2014 at 12:21 AM, Pascal VINCENT  
>>> wrote:
>>>
 Hi,

 I'm new to elasticsearch. My usecase is to load a csv file containing 
 some agencies with geo location, each lines are like :

 id;label;address;zipcode;city;region;*latitude*;*longitude*;(and some 
 others fields)+

 I'm using the csv river plugin to index the file.

 My mapping is :

 {
   "office": {
 "properties": {

 *(first fields omitted...)*

   "*latitude*": {
 "type": "double",
   },
   "*longitude*": {
 "type": "double",
   },
   "*location*": {
 "type": "geo_point",
 "lat_lon": "true"
   }
 }  
 }

 I'd like to index the location .lon and .lat value from the latitude 
 and longitude fields. I tried the copy_to function with no success :
   "latitude": {
 "type": "double",
 "copy_to": "location.lat"
   },
   "longitude": {
 "type": "double",
 "copy_to": "location.lon"
   },

 Is there any way to feed the "location" property from latitude and 
 longitude fields at indexation ?

 My point is that I don't want to modify the input csv file to adapt it 
 to the GeoJSON format (i.e concat lat and lon in one field in the csv 
 file).

 Thank you for any hints.

 Pascal.

  -- 
 You received this message because you are subscribed to the Google 
 Groups "elasticsearch" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to elasticsearc...@googlegroups.com.

 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/elasticsearch/6e12ced7-5b1a-4142-93d1-a3d22d7138a2%40googlegroups.com
  
 
 .
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>  -- 
>>> You received this message because you are subscribed to a topic in the 
>>> Google Groups "elasticsearch" group.
>>> To unsubscribe from this topic, visit 
>>> https://groups.google.com/d/topic/elasticsearch/QaI1fj74RlM/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to 
>>> elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-uHKT74qVbDT%3D8qg5Cv4vH0y%3DOzC8hGyO2uq_sY3sJ8g%40mail.gmail.com
>>>  
>>> 
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elast

Re: Elasticsearch 1.4.4-1 with Shield 1.0.1 on CentOS 6.6 - Authentication issue when running as service vs bin/elasticsearch.

2015-04-01 Thread Lobna Tonn


How to delete this shield plugin pleaase? I'm facing the same error since I 
have my ES running as a service !
On Monday, February 23, 2015 at 4:28:32 PM UTC+1, Jason wrote:
>
> I am able to run bin/elasticsearch and authenticate just fine but whenever 
> I run elasticsearch as a service, I consistently receive the following 
> error:
>
> {
>   "error" : "AuthenticationException[unable to authenticate user 
> [rdeniro] for REST request [/idx1?pretty]]",
>   "status" : 401
> }
>
> I made sure that the $ES_HOME/config/shield/ dir and files were all owned 
> by ES user and tried to dig down to see how the two startup methods differ. 
> Ultimately, it came down to something in 
> /etc/elasticsearch/elasticsearch.yml by comparing the two runtime 
> environments, but haven't been able to make much headway from there.
>
> Anyone have any thoughts on this? Thanks in advance for any help.
>
> Cheers,
> Jason
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/6a4699c1-fb70-4d1c-afe8-db6daa3d3406%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to get aggregations working in Elasticsearch Spark adapter ?

2015-04-01 Thread michele crudele

Thanks,

when is the 2.1 release coming?

Another question, which I think is related to this one btw... I was able to 
run this piece of code using facets:

val q5 = """
|{
|  "query": {
|"match": { "_all": "error"}},
|  "facets":{
|"appName": {"terms": {"field": "appName"}},
|"sourceName": {"terms": {"field": "sourceName"}}}
|}
""".stripMargin


println("Query: " + q5)

val rdd = sc.esRDD("logs/app", q5);

What I get from the rdd are tuples (docID, Map[of the field=value]). Should 
I also expect to find facets ? If so, how do I get them ?


Il giorno mercoledì 1 aprile 2015 12:02:20 UTC+2, Costin Leau ha scritto:
>
> The short answer is that the connector relies on scan/scroll search for 
> its core functionality. And with aggs it needs 
> to switch the way it queries the cluster to a count search. 
> This is the last major feature that needs to be addressed before the 2.1 
> release. There's also an issue for it raised 
> here [1] which you can track. 
>
> Cheers, 
>
> [1] https://github.com/elastic/elasticsearch-hadoop/issues/276 
>
> On 4/1/15 12:53 PM, michele crudele wrote: 
> > 
> > I have ES, Spark, and ES hadoop adapter installed on my laptop. I wrote 
> a simple scala notebook to test ES adapter. 
> > Everything was fine until I started thinking at more sophisticated 
> features. This is the snippet that drives me crazy: 
> > 
> > %AddJar 
> file:///tools/elasticsearch-hadoop-2.1.0.Beta3/dist/elasticsearch-hadoop-2.1.0.BUILD-SNAPSHOT.jar
>  
>
> > %AddJar 
> file:///tools/elasticsearch-hadoop-2.1.0.Beta3/dist/elasticsearch-spark_2.10-2.1.0.BUILD-SNAPSHOT.jar
>  
>
> > 
> > import org.elasticsearch.spark.rdd._ 
> > 
> > val q2 = """{ 
> >  |"query" : { "term": { "appName": "console" } }, 
> >  |"aggregations": { 
> >  |  "unusual": { 
> >  |"significant_terms": {"field": "pathname"} 
> >  |  } 
> >  |} 
> > |}""".stripMargin 
> > 
> > val res = sc.esRDD("logs/app", q2); 
> > 
> > println("Matches: " + res.count()) 
> > 
> > 
> > When I run the code I get this exception: 
> > 
> > Name: org.apache.spark.SparkException 
> > Message: Job aborted due to stage failure: Task 2 in stage 15.0 failed 1 
> times, most recent failure: Lost task 2.0 in stage 15.0 (TID 58, 
> localhost): org.apache.spark.util.TaskCompletionListenerException: 
> SearchPhaseExecutionException[Failed to execute phase [init_scan], all 
> shards failed; shardFailures {[N1R-UlgOQCGXCFCtbJ3sBQ][logrecords][2]: 
> ElasticsearchIllegalArgumentException[aggregations are not supported with 
> search_type=scan]}] 
> > at 
> org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:76) 
>
> > at org.apache.spark.scheduler.Task.run(Task.scala:58) 
> > at 
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:200) 
> > at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  
>
> > at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  
>
> > at java.lang.Thread.run(Thread.java:745) 
> > 
> > 
> > "aggregations are not supported with search_type=scan", which is fine. 
> > The question is: how do I set search_type to the right value (e.g. 
> count) in the sc.esRDD() call? 
> > I tried several places in the q2 json with no success and I was not able 
> to find an answer through 
> > the documentation. I would appreciate any help. 
> > 
> > However, I see a possible inconsistency with the behaviour of the ES API 
> used directly via cURL. 
> > The command with the same query above, and without any setting about 
> search_type works correctly: 
> > 
> > curl 'localhost:9200/logs/app/_search?pretty' -d'{"query" : { "term": { 
> "appName": "console" } }, 
> > "aggregations": { "unusual": { "significant_terms": {"field": 
> "pathname"} }}}' 
> > 
> > returns hits:{} and aggregations:{}. Why the Spark integration does not 
> work the same ? 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups "elasticsearch" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to 
> > elasticsearc...@googlegroups.com   elasticsearch+unsubscr...@googlegroups.com >. 
> > To view this discussion on the web visit 
> > 
> https://groups.google.com/d/msgid/elasticsearch/d044d380-a4b2-4d22-8990-60f318f7601a%40googlegroups.com
>  
> > <
> https://groups.google.com/d/msgid/elasticsearch/d044d380-a4b2-4d22-8990-60f318f7601a%40googlegroups.com?utm_medium=email&utm_source=footer>.
>  
>
> > For more options, visit https://groups.google.com/d/optout. 
>
> -- 
> Costin 
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/63f36dfd-fdf7-46a8-b092-2df293b3

alias not indexing all documents

2015-04-01 Thread Stefanie

Hi,

I have an alias with filters that does not seem to be indexing any new 
documents.  When I run the filter directly on the index I get the correct 
results but when I call the alias the new documents are not being pulled.  

This only seems to be happening with one alias as all of my other aliases 
that work the same way don't seem to be having a problem.  I have tried 
deleting the alias and recreating it but this didn't fix the issue.

Does anyone know why this might be happening?

Thank you,
Stefanie

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/48a5267d-7fa4-4d6e-bacb-54b93775955a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Problem with ES- Shield

I wanted to add authentication to my ES, thus I downloaded the shield :
https://www.elastic.co/downloads/shield

Then I've tried to add a user to give him the admin role as it's mentionned 
here :
http://www.elastic.co/guide/en/shield/current/getting-started.html
(I've added it with root, cause I start my es with root..)

But when I try to do so, an error occure saying that there is a syntax 
error in the 38 line "the elef line" 
elef[$ES_INCLUDE] bla bla
I've tried many time : 
bin/shield/esusers useradd omar -p omar -r admin
bin/shield/esusers useradd esomar -p omar -r admin
bin/shield/esusers useradd es_omar -p omar -r admin
bin/shield/esusers useradd ES_omar -p omar -r admin
bin/shield/esusers useradd omar  -r admin
bin/shield/esusers useradd esomar  -r admin
bin/shield/esusers useradd es_omar  -r admin
bin/shield/esusers useradd ES_omar  -r admin

But the same error kept on jumping ! 

I said enough is enough and wanted to keep on doing wwhat I was doing, 
tried to run Kibana, and this is what happened !


I've then tried to open my ES marvel plugin, but there is now an 
authentication box 



I've tried ALL the combinations above, but none of them seemed to work !!

I'm really preoccupied by this note : 
To ensure that Elasticsearch can read the user and role information at 
startup, run esusers useradd as the same user you use to run Elasticsearch. 
Running the command as root or some other user will update the permissions 
for the users and users_roles files and prevent Elasticsearch from 
accessing them.

The thing is that I start my es and kibana with the root..
Please HELP !! 
ᐧ

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/0841dff2-a46e-4762-a34f-9e4d2cf94885%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Settings Exception with repository-hdfs plugin

2015-04-01 Thread Scott E Lundgren

Thank you for linking to the yml file in the master branch as an example. I did 
have spaces but I had forgotten a colon after repositories!

- SL

> On Apr 1, 2015, at 6:37 AM, Costin Leau  wrote:
> 
> You probably have a syntax error (make sure you are using spaces and NOT tabs 
> in your yml file). I just did a quick check and it worked with the latest ES.
> As inspiration you can try the configuration used within the test suite 
> (checked every night) available here [1]. Additionally you can register the 
> repository dynamically through the REST API as documented here [2]. This 
> makes it easy to check various settings and see what works and doesn't.
> 
> Hope this helps,
> 
> [1] 
> https://github.com/elastic/elasticsearch-hadoop/blob/master/repository-hdfs/src/itest/resources/elasticsearch.yml
>  
> 
> [2] 
> http://www.elastic.co/guide/en/elasticsearch/reference/1.5/modules-snapshots.html
>  
> 
> 
> On 3/30/15 7:31 PM, Scott E Lundgren wrote:
>> Can someone point out what's wrong with my configuration of the 
>> repository-hdfs plugin in my elasticsearch.yml ? I'm
>> using Elasticsearch v1.3.4, repository-hdfs v2.0.2. The relevant part of 
>> elasticsearch.yml:
>> 
>> repositories
>>   hdfs:
>> path: "/user/jce/archive"
>> 
>> I think I'm following the configuration documentation
>> at https://github.com/elastic/elasticsearch-hadoop/tree/2.0/repository-hdfs 
>> but I'm seeing errors of:
>> 
>> - SettingsException[Failed to load settings from 
>> [file:/etc/elasticsearch/elasticsearch.yml]]
>> ElasticsearchParseException[malformed, expected settings to start with 
>> 'object', instead was [VALUE_STRING]]
>> 
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to
>> elasticsearch+unsubscr...@googlegroups.com 
>>  
>> > >.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/b1aeb6ee-697a-4bbb-b5f4-d501785cef1f%40googlegroups.com
>>  
>> 
>> >  
>> >.
>> For more options, visit https://groups.google.com/d/optout 
>> .
> 
> -- 
> Costin
> 
> -- 
> You received this message because you are subscribed to a topic in the Google 
> Groups "elasticsearch" group.
> To unsubscribe from this topic, visit 
> https://groups.google.com/d/topic/elasticsearch/goTVSOtXuXQ/unsubscribe 
> .
> To unsubscribe from this group and all its topics, send an email to 
> elasticsearch+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/551BCA7E.7060204%40gmail.com 
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/F8E6B900-D6A2-4150-9C82-45522C9BA416%40scottelundgren.com.
For more options, visit https://groups.google.com/d/optout.

How to perform Sum of a field to a group by data....

2015-04-01 Thread Muddadi Hemaanusha

Hi Team,

I would like to get the exact result of an SQL query 

select sum(Amount) from Emp group by (Name) Here If I take an example 
like

id  Name   amount
1   A1100
2   A2 200
3   A3 300
4   A1 200
5   A2 300

So for the above query the result will be Sum(Amount) by grouping the names 
A1, A2, A3 and for A1--> sum(100+200=300), A2-->sum(200+300=500), 
A3-->sum(300)...

For this query I have used the following equivalent query using 
Aggregations as shown below:

GET ABCD/_search
{
"size": 0, 
   
  "aggs": {
"group_trans": {
  "terms": {
"field": "trans"
  },
  "aggs": {
"sum_ordamt":{
"sum":{
"field":"ordamt"
}
}
}
  }
}
  }

Am getting the result as:

"aggregations": {
  "group_trans": {
 "doc_count_error_upper_bound": 81,
 "sum_other_doc_count": 1371988,
 "buckets": [
{
   "key": "7ce2d738-5549-4da2-aed0-db939f3a9188",
   "doc_count": 282,
   "sum_ordamt": {
  "value": 0
   }
},
{
   "key": "8f4b187a-f321-4d25-bbc0-a9e78f5810b8",
   "doc_count": 230,
   "sum_ordamt": {
  "value": 0
   }
},
{
   "key": "ca54db4c-620b-4650-bef6-350e83a0ca2c",
   "doc_count": 170,
   "sum_ordamt": {
  "value": 0
   }
}
.
.
.
 ]
  }
   }
}   .Here Am getting sum 
value==0, May I know is there any way to Make a query relevant to the SQL 
query given above for to give the sum depending on the group by names... 

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/ab20bcf5-d573-431b-bb9e-7c1207109988%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Kibana 4 detects geodata property but doesn't display anything in a geomap visualisation

2015-04-01 Thread Vagif Abilov

Hello,

I have created an Elasticsearch index from a data set containing geodata. I
have set up mapping for the data. Then I tried to create Kibana
visualisation using this data set. Kibana detects the geodata property but
finds no result even though there plenty of. Then I ran a test on another
data set with different and much simpler layout, and Kibana properly
visualised geodata.

Here's the sample that works:

"location": {
"lat": 56.290525,
"lon": -30.163298
},

and this is its mapping:

"location": {
"type": "geo_point",
"lat_lon": true,
"geohash": true
}

And this one doesn't work:

"groupOfLocations": {
"@type": "Point",
"locationForDisplay": {
"lat": 59.21232,
"lon": 9.603803
}
}

And this is its mapping:

{
... // nested type

"locationForDisplay": {
"type": "geo_point",
"lat_lon": true,
"geohash": true
}
...
}

There are only two things that are different between working and
non-working versions:
1. The one that works has a JSON element called "location" while the other
one is called "locationForDisplay"
2. The one that works has a JSON element ("location") as a top level
element, while in the other one it's an element in the nested type.

Apart from these two differences (which I believe shouldn't mean anything)
I can't find anything else. What can make Kibana fail?

Thanks in advance

Vagif

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/91aff9ac-83bf-49b7-94a2-48ebeeb766f8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: mlockall

2015-04-01 Thread Jakub Liska

I have the same problem with not only mlockall 


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1f990d34-9ec9-41e3-8d79-e6bb53ac0428%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

URGENT HELP : can't connect to my ES and Kibana after installing shield !

Hi everyone, 

I've installed the shield to add authentication to my ES ! 

When I've tried to add a user with an admin role, an error has occured 
saying that there is a syntax error in the elif[$ES_INCLUDE]

(I've created the user with the root since I start my ES with the root user 
!)

Now, when I try to run Kibana, this error jumps up : 


And when I try to access to my Marvel plugin, a box showing authentication 
required jumps to my face ! when I enter the user I've created (with the 
error I mentionned) , I can't access to it !!!


What to do please??! I'm in a huge hurry !


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f9984301-7077-4893-8e13-63900d28cf82%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: unable to get script_fields to work

2015-04-01 Thread Gerald DeConto

Unfortunately no.  Had to move on and skip scripts



On 1:27pm, Mar 31, 2015, at 1:27pm, Julia Neidert  wrote:
>I'm having the same problem. Were you able to resolve this? Thanks!
>
>
>On Thursday, February 26, 2015 at 10:38:26 AM UTC-8, Gerald DeConto
>wrote:
>>
>> fyi:
>>
>> my config files (elasticsearch.yml and logging.yml) are 
>> in /etc/elasticsearch.  the script_fields documentation indicated
>that I 
>> put the test.groovy in "config/scripts" folder, so I assumed my
>location is 
>> correct as I am unable to find a "config" folder in any elasticsearch
>folder
>>
>
>-- 
>You received this message because you are subscribed to a topic in the
>Google Groups "elasticsearch" group.
>To unsubscribe from this topic, visit
>https://groups.google.com/d/topic/elasticsearch/4GOnbUeWw6c/unsubscribe.
>To unsubscribe from this group and all its topics, send an email to
>elasticsearch+unsubscr...@googlegroups.com.
>To view this discussion on the web visit
>https://groups.google.com/d/msgid/elasticsearch/1b71e8f1-7cb7-41d2-8587-b6a5f88e5ebb%40googlegroups.com.
>For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/649a33fd-1f7d-40ba-98d0-2236514f9a47%40email.typeapp.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to make unstructured data available in Hadoop for analysis with Elasticsearch?

2015-04-01 Thread 'BEN SALEM Omar' via elasticsearch

I've removed the shield plugin with a yum remove shield but the error still
persist !
ᐧ

On Wed, Apr 1, 2015 at 3:42 PM, BEN SALEM Omar 
wrote:

> Costin HELP !!
>
> I've installed shield and tried to add an admin user, but when I do so, an
> error jumps saying that there is an error in the elif line.
> But now, if I try to run Kibana this is what happens !!! :
>
> Help plese !!
> 
> ᐧ
>
> On Wed, Apr 1, 2015 at 12:23 PM, BEN SALEM Omar 
> wrote:
>
>> Thank you for your time and help Costin !
>>
>>
>>
>>
>> ᐧ
>>
>> On Wed, Apr 1, 2015 at 11:53 AM, Costin Leau 
>> wrote:
>>
>>> Depends on what you want to do with the data - you need some type of
>>> analyzer to interpret your data typically depending on what type of queries
>>> you are making
>>> since otherwise it's just bytes.
>>> Your images need to be converted into something - RGB signature, shapes,
>>> etc... Same for a text file - it in English, Spanish, Japanese? Is it a
>>> pdf, xls - again ES needs to understand the format and there are multiple
>>> plugins for that. Even if you don't know what you are looking, you need to
>>> break down the blob of bytes into 'something' that can be searched on.
>>>
>>> As for Kibana, it's all JS. Some are fine with embedding it into their
>>> own app or exposing directly, some prefer more control and end up modifying
>>> it or putting their own controls in front that apply filtering or enforce a
>>> read-only view. The Kibana team is working on enhancing this even further
>>> so it's best to talk to them about specific technical issues which I'm
>>> afraid I can't answer.
>>>
>>> Hope this helps,
>>>
>>> On 4/1/15 12:33 PM, 'BEN SALEM Omar' via elasticsearch wrote:
>>>
 Hi Costin,

 I've already done that and here I am using Kibana to dashboards and so
 on(I have a question for you I'll ask a little
 bit later)

 But, my question now is how to make UNstructured data available for
 such analytics?
 Let's suppose I have images or textfile that I want to analyse? How
 make it available in a structured way in hive to do
 the required steps untill loading it into my ES cluster and perform my
 analytics?

 The second question is , I have created some Dashboards with Kibana;
 now what I want to do, is giving my dashboards to
 the client I'm working with/for to to do his analytics without letting
 him see what's going on the ES cluster.
 What's the easiest way to do this?
 Thanks,
 ᐧ

 On Wed, Apr 1, 2015 at 11:20 AM, Costin Leau >>> > wrote:

 If your data is in Hadoop, you would index it in Elasticsearch
 (through the es-hadoop connector) and then run your
 queries against it (either from Hadoop through the same connector)
 or through other tools (like Kibana) outside Hadoop.

 There are plenty of blog posts, documentations, presentations and
 videos on the topic so I recommend spending
 sometime time with them to better understand what options suite
 your use case.

 On 4/1/15 12:12 PM, BEN SALEM Omar wrote:

 Hi everyone,
 I'm wondering, it possible to make analytics on unstructured
 data with Elasticsearch?
 If so? how?
 Thanks,
 Omar,

 --
 You received this message because you are subscribed to the
 Google Groups "elasticsearch" group.
 To unsubscribe from this group and stop receiving emails from
 it, send an email to
 elasticsearch+unsubscribe@__googlegroups.com >>> elasticsearch%2bunsubscr...@googlegroups.com>
 >> elasticsearch%2bunsubscr...@googlegroups.com>>.
 To view this discussion on the web visit
 https://groups.google.com/d/__msgid/elasticsearch/08e7617c-_
 _9e72-40a6-a780-7f632fee6d47%__40googlegroups.com

 >.
 For more options, visit https://groups.google.com/d/__optout <
 https://groups.google.com/d/optout>.

 --
 Costin

 --
 You received this message because you are subscribed to a topic in
 the Google Groups "elasticsearch" group.
 To unsubscribe from this topic, visit
 https://groups.google.com/d/__topic/elasticsearch/__
 Md6pwsOI2Dw/unsubscribe
 .
 To unsubscribe from th

Re: How to make unstructured data available in Hadoop for analysis with Elasticsearch?

2015-04-01 Thread 'BEN SALEM Omar' via elasticsearch

Costin HELP !!

I've installed shield and tried to add an admin user, but when I do so, an
error jumps saying that there is an error in the elif line.
But now, if I try to run Kibana this is what happens !!! :

Help plese !!

ᐧ

On Wed, Apr 1, 2015 at 12:23 PM, BEN SALEM Omar 
wrote:

> Thank you for your time and help Costin !
>
>
>
>
> ᐧ
>
> On Wed, Apr 1, 2015 at 11:53 AM, Costin Leau 
> wrote:
>
>> Depends on what you want to do with the data - you need some type of
>> analyzer to interpret your data typically depending on what type of queries
>> you are making
>> since otherwise it's just bytes.
>> Your images need to be converted into something - RGB signature, shapes,
>> etc... Same for a text file - it in English, Spanish, Japanese? Is it a
>> pdf, xls - again ES needs to understand the format and there are multiple
>> plugins for that. Even if you don't know what you are looking, you need to
>> break down the blob of bytes into 'something' that can be searched on.
>>
>> As for Kibana, it's all JS. Some are fine with embedding it into their
>> own app or exposing directly, some prefer more control and end up modifying
>> it or putting their own controls in front that apply filtering or enforce a
>> read-only view. The Kibana team is working on enhancing this even further
>> so it's best to talk to them about specific technical issues which I'm
>> afraid I can't answer.
>>
>> Hope this helps,
>>
>> On 4/1/15 12:33 PM, 'BEN SALEM Omar' via elasticsearch wrote:
>>
>>> Hi Costin,
>>>
>>> I've already done that and here I am using Kibana to dashboards and so
>>> on(I have a question for you I'll ask a little
>>> bit later)
>>>
>>> But, my question now is how to make UNstructured data available for such
>>> analytics?
>>> Let's suppose I have images or textfile that I want to analyse? How make
>>> it available in a structured way in hive to do
>>> the required steps untill loading it into my ES cluster and perform my
>>> analytics?
>>>
>>> The second question is , I have created some Dashboards with Kibana; now
>>> what I want to do, is giving my dashboards to
>>> the client I'm working with/for to to do his analytics without letting
>>> him see what's going on the ES cluster.
>>> What's the easiest way to do this?
>>> Thanks,
>>> ᐧ
>>>
>>> On Wed, Apr 1, 2015 at 11:20 AM, Costin Leau >> > wrote:
>>>
>>> If your data is in Hadoop, you would index it in Elasticsearch
>>> (through the es-hadoop connector) and then run your
>>> queries against it (either from Hadoop through the same connector)
>>> or through other tools (like Kibana) outside Hadoop.
>>>
>>> There are plenty of blog posts, documentations, presentations and
>>> videos on the topic so I recommend spending
>>> sometime time with them to better understand what options suite your
>>> use case.
>>>
>>> On 4/1/15 12:12 PM, BEN SALEM Omar wrote:
>>>
>>> Hi everyone,
>>> I'm wondering, it possible to make analytics on unstructured
>>> data with Elasticsearch?
>>> If so? how?
>>> Thanks,
>>> Omar,
>>>
>>> --
>>> You received this message because you are subscribed to the
>>> Google Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from
>>> it, send an email to
>>> elasticsearch+unsubscribe@__googlegroups.com >> elasticsearch%2bunsubscr...@googlegroups.com>
>>> > elasticsearch%2bunsubscr...@googlegroups.com>>.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/__msgid/elasticsearch/08e7617c-_
>>> _9e72-40a6-a780-7f632fee6d47%__40googlegroups.com
>>> >> 9e72-40a6-a780-7f632fee6d47%40googlegroups.com>
>>> >> __9e72-40a6-a780-7f632fee6d47%__40googlegroups.com?utm_
>>> medium=__email&utm_source=footer
>>> >> 9e72-40a6-a780-7f632fee6d47%40googlegroups.com?utm_medium=
>>> email&utm_source=footer>>.
>>> For more options, visit https://groups.google.com/d/__optout <
>>> https://groups.google.com/d/optout>.
>>>
>>> --
>>> Costin
>>>
>>> --
>>> You received this message because you are subscribed to a topic in
>>> the Google Groups "elasticsearch" group.
>>> To unsubscribe from this topic, visit https://groups.google.com/d/__
>>> topic/elasticsearch/__Md6pwsOI2Dw/unsubscribe
>>> >> Md6pwsOI2Dw/unsubscribe>.
>>> To unsubscribe from this group and all its topics, send an email to
>>> elasticsearch+unsubscribe@__googlegroups.com
>>> .
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/__msgid/elasticsearch/551BB843._
>>>

403 from kibana when searching closed indices

2015-04-01 Thread Derry O' Sullivan

Hi all,

Am using the standard ELK stack and am having an issue with kibana 
currently.

I close one of my indices (e.g. logstash-2015.03.30). When i do a search in 
kibana for data involving that index, i get a 403 (IndexClosedException). 
 I would have expected Kibana/Elasticsearch to know the index is closed and 
not to use in in search.

Is there anyway to tell kibana to ignore that index in search? Would 
deleting the index be an alternative? 

Derry

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/2f6f60d5-fbf0-4ad6-be8e-d4769e62c618%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Some of elasticsearch.yml configuration is not taking effect

2015-04-01 Thread Jakub Liska

Hey,

I've asked this question on SO 
,
 
I have a problem, that settings from elasticsearch.yml like:

 indices.memory.index_buffer_size: 50%

Are not reflected in GET _cluster/settings as they should, right? Even 
though others like cluster-name or discovery are...

What am I doing wrong here? I tried to restart ES with multiple ways of 
setting it up, nothing helps. Thank you

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/a0be7f56-da32-4b08-b304-3e99f2f3c9cf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to Backup Cluster by Time Scheduled:

2015-04-01 Thread Pradeep Reddy

You can set up a cron job to run a script like this every day

#!/bin/sh
> curl -XPUT "localhost:9200/_snapshot/my_s3_repository/$(date 
> +"%Y.%m.%d.%H.%M.%S")?wait_for_completion=true?pretty"



On Wednesday, April 1, 2015 at 5:17:57 PM UTC+5:30, James Crone wrote:
>
> I have create a backup directory using FS Technique.
>
>  
>
>  PUT _snapshot/databackup {
>"type": "fs", 
>"settings": {
>   "location": "/mount/backups/my_backup" 
>}}
>
> Then:
>
>  PUT _snapshot/my_backup/snapshot_2 {
>"indices": "completedata" }
>
>
> Its working.But when my database table update or insert new records backup 
> not update automatically.How Can i Do this auto
> backup after 1 day or by defining time. 
>
> Thanks in advance.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d0b746f4-4587-44d5-872a-ccec64e41cdd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

How to Backup Cluster by Time Scheduled:

2015-04-01 Thread James Crone

I have create a backup directory using FS Technique.

 

 PUT _snapshot/databackup {
   "type": "fs", 
   "settings": {
  "location": "/mount/backups/my_backup" 
   }}

Then:

 PUT _snapshot/my_backup/snapshot_2 {
   "indices": "completedata" }


Its working.But when my database table update or insert new records backup not 
update automatically.How Can i Do this auto
backup after 1 day or by defining time. 

Thanks in advance.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/3aa4d6ea-a0c5-427c-973d-90ea292700c8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Very regular disconnect and recover - every 2 hours

2015-04-01 Thread Neil Andrassy

Further to this, the cluster that's failing has far more shards than the
one that stays up. We have a number of daily date stamped indexes, each
retaining 90 days of data amounting to 7500+ shards.

Looking through github, it looks like some underlying changes might have
impacted the scalability of clusters with larger numbers of shards (*[STORE]
Cache fileLength for fully written files #9683*, *ES 1.4.2 random node
disconnect #9212* and *[STORE] Add simple cache for StoreStats #9709 *seem
related). Looks like there are some improvements in 1.5.0 (particulalry
#9709). We're in the process of testing the upgrade (although there's a
regression too that's holding us up slightly - exists filters don't work on
indexes created prior to 1.3.0 *Mappings: Fix _field_names to be disabled
on pre 1.3.0 indexes #10268* - fixed in 1.5.1 when it arrives). We have
also attempted to reduce the number of shards where possible as a short
term workaround. We also found that CPU usage is VERY high on the nodes
that stayed in cluster during recovery - we've dropped concurrent shard
recoveries to 1 (from the default of 3) to ease this situation.

We're also considering weekly or even monthly rolling indexes to reduce
shard count but that's a bigger code change in our application.

Watch this space...

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/830a91b3-ba2e-40b2-a166-8e0e14a4f99d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Snapshot is stuck in IN_PROGRESS

2015-04-01 Thread Pradeep Reddy

I am using cloud aws plugin, Snapshots are taken directly to S3. 
It's a 3 node cluster.

*ES config:*
cloud:
aws:
access_key: xxx
secret_key: yyy
protocol: http
region: us-west-1
discovery.type: ec2
discovery.ec2.groups: elasticsearch
repositories:
s3:
bucket: "xxx"
region: "us-west-1"

When I run this 
 

> curl -XGET 
> 'localhost:9200/_snapshot/my_s3_repository/2015.03.11.06.04.39/_status?pretty'


Part of the output is 
  "snapshots" : [ {
"snapshot" : "2015.03.11.06.04.39",
"repository" : "my_s3_repository",
"state" : "ABORTED",
"shards_stats" : {
  "initializing" : 0,
  "started" : 0,
  "finalizing" : 0,
  "done" : 17,
  "failed" : 3,
  "total" : 20
},

When I run 

> curl localhost:9200/_snapshot/_all?pretty

it shows IN_PROGRESS
 {
"snapshot" : "2015.03.11.06.04.39",
"indices" : [ "playground_video", "playground_admin", 
"playground_profile", "playground_feed" ],
"state" : "IN_PROGRESS",
"start_time" : "2015-03-11T06:04:39.044Z",
"start_time_in_millis" : 1426053879044,
"failures" : [ ],
"shards" : {
  "total" : 0,
  "failed" : 0,
  "successful" : 0
}
  }

 curl -XDELETE 
> 'localhost:9200/_snapshot/my_s3_repository/2015.03.11.06.04.39' 

this operation never completes. I left it running for a day..still no use..

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4a34a807-d596-4ca7-b90b-f709f52d624b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: mlockall

2015-04-01 Thread Cheten Dev

Hi,

I don't know  what config sections you are asking about, but here you can
see information about  two nodes

i have uncommented bootstrap:mlockall = true
i have tried ulimit -l unlimited
i also tried the following ./bin/elasticsearch -Djna.tmpdir=/some path/

but no success


root@ubuntu:~/chetan/elasticsearch-1.5.0/bin# curl
localhost:9200/_nodes/process ?pretty
{
  "cluster_name" : "CWESUBUNTU",
  "nodes" : {
"rD-BkWRnQ-uE5_VoWqStiA" : {
  "name" : "ubuntu200",
  "transport_address" : "inet[/192.168.1.200:9300]",
  "host" : "ubuntu",
  "ip" : "192.168.1.200",
  "version" : "1.5.0",
  "build" : "5448160",
  "http_address" : "inet[/192.168.1.200:9200]",
  "process" : {
"refresh_interval_in_millis" : 1000,
"id" : 12999,
"max_file_descriptors" : 9,
"mlockall" : false
  }
},
"vGHvM7ncQ9id36ZcpSIFDQ" : {
  "name" : "ubuntu202",
  "transport_address" : "inet[/192.168.1.202:9300]",
  "host" : "ubuntu02",
  "ip" : "192.168.1.202",
  "version" : "1.5.0",
  "build" : "5448160",
  "http_address" : "inet[/192.168.1.202:9200]",
  "process" : {
"refresh_interval_in_millis" : 1000,
"id" : 12234,
"max_file_descriptors" : 9,
"mlockall" : true
  }
}
  }
}


On Wed, Apr 1, 2015 at 5:28 AM, Mark Walkom  wrote:

> Can you show us the relevant config sections from the nodes that don't
> have mlockall set?
>
> On 31 March 2015 at 17:40, Cheten Dev  wrote:
>
>> Hi,
>>
>> here is the info
>>
>> Distributor ID: Ubuntu
>> Description:Ubuntu 14.04.1 LTS
>> Release:14.04
>> Codename:   trusty
>>
>> I have tried to set mlockall = true on two server . i am able to
>> successfully do it on one
>> but not able to do it on other server
>>
>>
>> On Tue, Mar 31, 2015 at 11:56 AM, Mark Walkom 
>> wrote:
>>
>>> Which linux?
>>>
>>> On 31 March 2015 at 16:16, Cheten Dev  wrote:
>>>
 Hi,

 I am on linux server with elasticsearch 1.5.0

 On Tue, Mar 31, 2015 at 8:07 AM, Mark Walkom 
 wrote:

> A bit more info would be useful.
>
> What version of ES, what OS?
>
> On 30 March 2015 at 22:24, Chetan Dev  wrote:
>
>> Hi,
>>
>> i have tried everything written on the document to set mlackall =
>> true  but
>> nothing seems working here
>>
>> what am i missing here ?
>>
>>
>> Thanks
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/5748d9fb-120a-4521-8caa-62b780ea0594%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/R41hW2QaL0w/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-92m9Qw4j4T8gwUV6%2BumkvbAOENTTsiupMurpGf8NZSw%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

  --
 You received this message because you are subscribed to the Google
 Groups "elasticsearch" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to elasticsearch+unsubscr...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/elasticsearch/CAG_NmL9T4dZeHUYpN0_5gwbfppL7%2BOJyNc6t3SQUpSe9N5oN2Q%40mail.gmail.com
 
 .
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>  --
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "elasticsearch" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/elasticsearch/R41hW2QaL0w/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to
>>> elasticsearch+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> ht

Re: Settings Exception with repository-hdfs plugin

You probably have a syntax error (make sure you are using spaces and NOT tabs in your yml file). I just did a quick
check and it worked with the latest ES.
As inspiration you can try the configuration used within the test suite (checked every night) available here [1].
Additionally you can register the repository dynamically through the REST API as documented here [2]. This makes it easy
to check various settings and see what works and doesn't.

Hope this helps,

[1]
https://github.com/elastic/elasticsearch-hadoop/blob/master/repository-hdfs/src/itest/resources/elasticsearch.yml
[2]
http://www.elastic.co/guide/en/elasticsearch/reference/1.5/modules-snapshots.html

On 3/30/15 7:31 PM, Scott E Lundgren wrote:

Can someone point out what's wrong with my configuration of the repository-hdfs
plugin in my elasticsearch.yml ? I'm
using Elasticsearch v1.3.4, repository-hdfs v2.0.2. The relevant part of
elasticsearch.yml:

repositories
hdfs:
path: "/user/jce/archive"

I think I'm following the configuration documentation
at https://github.com/elastic/elasticsearch-hadoop/tree/2.0/repository-hdfs but
I'm seeing errors of:

- SettingsException[Failed to load settings from
[file:/etc/elasticsearch/elasticsearch.yml]]
ElasticsearchParseException[malformed, expected settings to start with
'object', instead was [VALUE_STRING]]

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
elasticsearch+unsubscr...@googlegroups.com
.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b1aeb6ee-697a-4bbb-b5f4-d501785cef1f%40googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.

--
Costin

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/551BCA7E.7060204%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to make unstructured data available in Hadoop for analysis with Elasticsearch?

2015-04-01 Thread 'BEN SALEM Omar' via elasticsearch

Thank you for your time and help Costin !




ᐧ

On Wed, Apr 1, 2015 at 11:53 AM, Costin Leau  wrote:

> Depends on what you want to do with the data - you need some type of
> analyzer to interpret your data typically depending on what type of queries
> you are making
> since otherwise it's just bytes.
> Your images need to be converted into something - RGB signature, shapes,
> etc... Same for a text file - it in English, Spanish, Japanese? Is it a
> pdf, xls - again ES needs to understand the format and there are multiple
> plugins for that. Even if you don't know what you are looking, you need to
> break down the blob of bytes into 'something' that can be searched on.
>
> As for Kibana, it's all JS. Some are fine with embedding it into their own
> app or exposing directly, some prefer more control and end up modifying it
> or putting their own controls in front that apply filtering or enforce a
> read-only view. The Kibana team is working on enhancing this even further
> so it's best to talk to them about specific technical issues which I'm
> afraid I can't answer.
>
> Hope this helps,
>
> On 4/1/15 12:33 PM, 'BEN SALEM Omar' via elasticsearch wrote:
>
>> Hi Costin,
>>
>> I've already done that and here I am using Kibana to dashboards and so
>> on(I have a question for you I'll ask a little
>> bit later)
>>
>> But, my question now is how to make UNstructured data available for such
>> analytics?
>> Let's suppose I have images or textfile that I want to analyse? How make
>> it available in a structured way in hive to do
>> the required steps untill loading it into my ES cluster and perform my
>> analytics?
>>
>> The second question is , I have created some Dashboards with Kibana; now
>> what I want to do, is giving my dashboards to
>> the client I'm working with/for to to do his analytics without letting
>> him see what's going on the ES cluster.
>> What's the easiest way to do this?
>> Thanks,
>> ᐧ
>>
>> On Wed, Apr 1, 2015 at 11:20 AM, Costin Leau > > wrote:
>>
>> If your data is in Hadoop, you would index it in Elasticsearch
>> (through the es-hadoop connector) and then run your
>> queries against it (either from Hadoop through the same connector) or
>> through other tools (like Kibana) outside Hadoop.
>>
>> There are plenty of blog posts, documentations, presentations and
>> videos on the topic so I recommend spending
>> sometime time with them to better understand what options suite your
>> use case.
>>
>> On 4/1/15 12:12 PM, BEN SALEM Omar wrote:
>>
>> Hi everyone,
>> I'm wondering, it possible to make analytics on unstructured data
>> with Elasticsearch?
>> If so? how?
>> Thanks,
>> Omar,
>>
>> --
>> You received this message because you are subscribed to the
>> Google Groups "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to
>> elasticsearch+unsubscribe@__googlegroups.com > elasticsearch%2bunsubscr...@googlegroups.com>
>>  elasticsearch%2bunsubscr...@googlegroups.com>>.
>> To view this discussion on the web visit
>> https://groups.google.com/d/__msgid/elasticsearch/08e7617c-_
>> _9e72-40a6-a780-7f632fee6d47%__40googlegroups.com
>> > 9e72-40a6-a780-7f632fee6d47%40googlegroups.com>
>> > __9e72-40a6-a780-7f632fee6d47%__40googlegroups.com?utm_
>> medium=__email&utm_source=footer
>> > 9e72-40a6-a780-7f632fee6d47%40googlegroups.com?utm_medium=
>> email&utm_source=footer>>.
>> For more options, visit https://groups.google.com/d/__optout <
>> https://groups.google.com/d/optout>.
>>
>> --
>> Costin
>>
>> --
>> You received this message because you are subscribed to a topic in
>> the Google Groups "elasticsearch" group.
>> To unsubscribe from this topic, visit https://groups.google.com/d/__
>> topic/elasticsearch/__Md6pwsOI2Dw/unsubscribe
>> > Md6pwsOI2Dw/unsubscribe>.
>> To unsubscribe from this group and all its topics, send an email to
>> elasticsearch+unsubscribe@__googlegroups.com
>> .
>> To view this discussion on the web visit
>> https://groups.google.com/d/__msgid/elasticsearch/551BB843._
>> _3040206%40gmail.com
>> > 3040206%40gmail.com>.
>> For more options, visit https://groups.google.com/d/__optout <
>> https://groups.google.com/d/optout>.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, se

Re: Elastichsearch CDH Hadoop Integration

2015-04-01 Thread Ravi sai kumar

Thank you so much Costin. I will configure it in locally.

On Wednesday, April 1, 2015 at 6:00:02 PM UTC+8, Costin Leau wrote:
>
> Elasticsearch has no problem handling tera-bytes of data and automatically 
> handles the data sharding, distribution and 
> replication across multiple machines. You don't need to have that space on 
> one node, ES automatically 'spreads' the data 
> across all its node - this the whole premise of scaling out. 
>
> However assuming you still don't want to do this do note that that HDFS is 
> not a file-system in the traditional sense, 
> you can't just point things to it. You can however expose it as a local 
> file system through NFS (check your Hadoop 
> documentation). However, do note that HDFS is significantly slower than a 
> local FS even in this case and in case of 
> writes might lead to inconsistent/corrupted data as mentioned here [1]. 
> That's not to say it should work rather it's an 
> unsupported scenario due to the many variables involved outside 
> Elasticsearch itself. 
>
> Cheers, 
>
> [1] https://github.com/elastic/elasticsearch/issues/9072 
>
>
> On 4/1/15 12:23 PM, Ravi sai kumar wrote: 
> > Thanks Costin Leau. 
> > My data will be in terabytes local mode doesnt suits my requirement may 
> be. If i set the hadoop path, 
> > Are there any changes we have to do for gateway.type? 
> > 
> > existing property is 
> > gateway.type: local 
> > 
> > 
> > On Wednesday, April 1, 2015 at 5:14:16 PM UTC+8, Costin Leau wrote: 
> > 
> > For best results and performance, point Elasticsearch to your local 
> storage. 
> > It's just like any other service (MySQL, Postgres, etc..) 
> > 
> > On 4/1/15 11:48 AM, Ravi sai kumar wrote: 
> > > I have the problem with elastichsearch integrating with hadoop. 
> > > I am new to elasticsearch. I have installed on CDH Single node 
> cluster. Please someone suggest me what should we keep at 
> > > path.data: /path to data 
> > > 
> > > that means, Indexed data location should be hdfs or local 
> directory? 
> > > 
> > > -- 
> > > You received this message because you are subscribed to the Google 
> Groups "elasticsearch" group. 
> > > To unsubscribe from this group and stop receiving emails from it, 
> send an email to 
> > >elasticsearc...@googlegroups.com   elasticsearch+unsubscr...@googlegroups.com  >. 
> > > To view this discussion on the web visit 
> > >
> https://groups.google.com/d/msgid/elasticsearch/b750ba78-360f-47af-8d8b-cb79e9e346f9%40googlegroups.com
>  
> > <
> https://groups.google.com/d/msgid/elasticsearch/b750ba78-360f-47af-8d8b-cb79e9e346f9%40googlegroups.com>
>  
>
> > > <
> https://groups.google.com/d/msgid/elasticsearch/b750ba78-360f-47af-8d8b-cb79e9e346f9%40googlegroups.com?utm_medium=email&utm_source=footer
>  
> > <
> https://groups.google.com/d/msgid/elasticsearch/b750ba78-360f-47af-8d8b-cb79e9e346f9%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>  
>
> > 
> > > For more options, visithttps://groups.google.com/d/optout <
> https://groups.google.com/d/optout>. 
> > 
> > -- 
> > Costin 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups "elasticsearch" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to 
> > elasticsearc...@googlegroups.com   elasticsearch+unsubscr...@googlegroups.com >. 
> > To view this discussion on the web visit 
> > 
> https://groups.google.com/d/msgid/elasticsearch/05d4ef8a-e1a4-47ef-ac26-7b4aada7cfac%40googlegroups.com
>  
> > <
> https://groups.google.com/d/msgid/elasticsearch/05d4ef8a-e1a4-47ef-ac26-7b4aada7cfac%40googlegroups.com?utm_medium=email&utm_source=footer>.
>  
>
> > For more options, visit https://groups.google.com/d/optout. 
>
> -- 
> Costin 
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5a35fd76-c47c-4c7d-8ba4-ebb3d82e940b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to get aggregations working in Elasticsearch Spark adapter ?

The short answer is that the connector relies on scan/scroll search for its core functionality. And with aggs it needs 
to switch the way it queries the cluster to a count search.
This is the last major feature that needs to be addressed before the 2.1 release. There's also an issue for it raised 
here [1] which you can track.


Cheers,

[1] https://github.com/elastic/elasticsearch-hadoop/issues/276

On 4/1/15 12:53 PM, michele crudele wrote:


I have ES, Spark, and ES hadoop adapter installed on my laptop. I wrote a 
simple scala notebook to test ES adapter.
Everything was fine until I started thinking at more sophisticated features. 
This is the snippet that drives me crazy:

%AddJar 
file:///tools/elasticsearch-hadoop-2.1.0.Beta3/dist/elasticsearch-hadoop-2.1.0.BUILD-SNAPSHOT.jar
%AddJar 
file:///tools/elasticsearch-hadoop-2.1.0.Beta3/dist/elasticsearch-spark_2.10-2.1.0.BUILD-SNAPSHOT.jar

import org.elasticsearch.spark.rdd._

val q2 = """{
 |"query" : { "term": { "appName": "console" } },
 |"aggregations": {
 |  "unusual": {
 |"significant_terms": {"field": "pathname"}
 |  }
 |}
|}""".stripMargin

val res = sc.esRDD("logs/app", q2);

println("Matches: " + res.count())


When I run the code I get this exception:

Name: org.apache.spark.SparkException
Message: Job aborted due to stage failure: Task 2 in stage 15.0 failed 1 times, 
most recent failure: Lost task 2.0 in stage 15.0 (TID 58, localhost): 
org.apache.spark.util.TaskCompletionListenerException: 
SearchPhaseExecutionException[Failed to execute phase [init_scan], all shards 
failed; shardFailures {[N1R-UlgOQCGXCFCtbJ3sBQ][logrecords][2]: 
ElasticsearchIllegalArgumentException[aggregations are not supported with 
search_type=scan]}]
at 
org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:76)
at org.apache.spark.scheduler.Task.run(Task.scala:58)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:200)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)


"aggregations are not supported with search_type=scan", which is fine.
The question is: how do I set search_type to the right value (e.g. count) in 
the sc.esRDD() call?
I tried several places in the q2 json with no success and I was not able to 
find an answer through
the documentation. I would appreciate any help.

However, I see a possible inconsistency with the behaviour of the ES API used 
directly via cURL.
The command with the same query above, and without any setting about 
search_type works correctly:

curl 'localhost:9200/logs/app/_search?pretty' -d'{"query" : { "term": { "appName": 
"console" } },
"aggregations": { "unusual": { "significant_terms": {"field": "pathname"} }}}'

returns hits:{} and aggregations:{}. Why the Spark integration does not work 
the same ?

--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to
elasticsearch+unsubscr...@googlegroups.com 
.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d044d380-a4b2-4d22-8990-60f318f7601a%40googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.


--
Costin

--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/551BC221.60400%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elastichsearch CDH Hadoop Integration

Elasticsearch has no problem handling tera-bytes of data and automatically handles the data sharding, distribution and 
replication across multiple machines. You don't need to have that space on one node, ES automatically 'spreads' the data 
across all its node - this the whole premise of scaling out.

However assuming you still don't want to do this do note that that HDFS is not a file-system in the traditional sense, 
you can't just point things to it. You can however expose it as a local file system through NFS (check your Hadoop 
documentation). However, do note that HDFS is significantly slower than a local FS even in this case and in case of 
writes might lead to inconsistent/corrupted data as mentioned here [1]. That's not to say it should work rather it's an 
unsupported scenario due to the many variables involved outside Elasticsearch itself.

Cheers,

[1] https://github.com/elastic/elasticsearch/issues/9072

On 4/1/15 12:23 PM, Ravi sai kumar wrote:

Thanks Costin Leau.
My data will be in terabytes local mode doesnt suits my requirement may be. If 
i set the hadoop path,
Are there any changes we have to do for gateway.type?

existing property is
gateway.type: local

On Wednesday, April 1, 2015 at 5:14:16 PM UTC+8, Costin Leau wrote:

For best results and performance, point Elasticsearch to your local storage.
It's just like any other service (MySQL, Postgres, etc..)

On 4/1/15 11:48 AM, Ravi sai kumar wrote:
> I have the problem with elastichsearch integrating with hadoop.
> I am new to elasticsearch. I have installed on CDH Single node cluster. 
Please someone suggest me what should we keep at
> path.data: /path to data
>
> that means, Indexed data location should be hdfs or local directory?
>
> --
> You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
email to
>elasticsearc...@googlegroups.com  
.
> To view this discussion on the web visit

>https://groups.google.com/d/msgid/elasticsearch/b750ba78-360f-47af-8d8b-cb79e9e346f9%40googlegroups.com

> 
>.

> For more options, visithttps://groups.google.com/d/optout 
.

--
Costin

--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to
elasticsearch+unsubscr...@googlegroups.com 
.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/05d4ef8a-e1a4-47ef-ac26-7b4aada7cfac%40googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.

--
Costin

--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/551BC184.3010803%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: exists filter broken on 1.5.0 with restored index?

2015-04-01 Thread Neil Andrassy

Seems like it also impacts "missing". Just checking that the patch will 
address that too? Looks like it will from the file changes in the patch 
code, but thought I'd better mention it, just in case! :)

On Thursday, 26 March 2015 18:31:44 UTC, Mads Martin Jørgensen wrote:
>
> Thanks for fixing!
>
> On Thursday, March 26, 2015 at 3:29:17 PM UTC+1, Igor Motov wrote:
>>
>> Thanks for checking. It's a bug, which should be fixed in 1.5.1 
>> https://github.com/elastic/elasticsearch/pull/10268
>>
>> On Wednesday, 25 March 2015 13:43:28 UTC-4, Mads Martin Jørgensen wrote:
>>>
>>> They're similar. The 1.5.0 cluster has "created" : "1000199", and the 
>>> 1.4.1 cluster also has "created" : "1000199"
>>>
>>> On Wednesday, March 25, 2015 at 4:45:30 PM UTC+1, Igor Motov wrote:

 Hi Mads Martin,

 Could you check the version that is returned when you run curl 
 "localhost:9200/my_own_index/_settings?pretty". The version will be in 

 "version" : {
   "created" : "XXX"
 }

 Could you compare it to the version that is returned by the same index 
 in the pre-1.5.0 cluster?

 Igor

 On Wednesday, 25 March 2015 09:27:03 UTC-4, Mads Martin Jørgensen wrote:
>
> Hello all,
>
> Just installed es-1.5.0 with cloud-aws-2.5.0 on a machine. Did a 
> restore of a snapshot made with es-1.4.1. All documents are there, but 
> the 
> exists filter seems broken. The query that used to return all documents 
> matching, now return 0 documents, even though the field exists when 
> reading 
> the documents.
>
> curl -XGET "http://localhost:9200/my_own_index/document/_search 
> "
>  
> -d'
>
> {
>
>"query": {
>
>   "constant_score": {
>
>  "filter": {
>
> "exists": {
>
>"field": "history"
>
> }
>
>  }
>
>   }
>
>}
>
> }'
>
>
> If we populate new documents, then the exists filter works just fine.
>
>
> Regards,
>
> Mads Martin
>


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4f4e4538-37e0-4da5-8761-1a65fbd74957%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to make unstructured data available in Hadoop for analysis with Elasticsearch?

Depends on what you want to do with the data - you need some type of analyzer to interpret your data typically depending
on what type of queries you are making

since otherwise it's just bytes.
Your images need to be converted into something - RGB signature, shapes, etc... Same for a text file - it in English,
Spanish, Japanese? Is it a pdf, xls - again ES needs to understand the format and there are multiple plugins for that.
Even if you don't know what you are looking, you need to break down the blob of bytes into 'something' that can be
searched on.

As for Kibana, it's all JS. Some are fine with embedding it into their own app or exposing directly, some prefer more
control and end up modifying it or putting their own controls in front that apply filtering or enforce a read-only view.
The Kibana team is working on enhancing this even further so it's best to talk to them about specific technical issues
which I'm afraid I can't answer.

Hope this helps,

On 4/1/15 12:33 PM, 'BEN SALEM Omar' via elasticsearch wrote:

Hi Costin,

I've already done that and here I am using Kibana to dashboards and so on(I
have a question for you I'll ask a little
bit later)

But, my question now is how to make UNstructured data available for such
analytics?
Let's suppose I have images or textfile that I want to analyse? How make it
available in a structured way in hive to do
the required steps untill loading it into my ES cluster and perform my
analytics?

The second question is , I have created some Dashboards with Kibana; now what I
want to do, is giving my dashboards to
the client I'm working with/for to to do his analytics without letting him see
what's going on the ES cluster.
What's the easiest way to do this?
Thanks,
ᐧ

On Wed, Apr 1, 2015 at 11:20 AM, Costin Leau mailto:costin.l...@gmail.com>> wrote:

If your data is in Hadoop, you would index it in Elasticsearch (through the
es-hadoop connector) and then run your
queries against it (either from Hadoop through the same connector) or
through other tools (like Kibana) outside Hadoop.

There are plenty of blog posts, documentations, presentations and videos on
the topic so I recommend spending
sometime time with them to better understand what options suite your use
case.

On 4/1/15 12:12 PM, BEN SALEM Omar wrote:

Hi everyone,
I'm wondering, it possible to make analytics on unstructured data with
Elasticsearch?
If so? how?
Thanks,
Omar,

>.
To view this discussion on the web visit

https://groups.google.com/d/__msgid/elasticsearch/08e7617c-__9e72-40a6-a780-7f632fee6d47%__40googlegroups.com

>.
For more options, visit https://groups.google.com/d/__optout
.

--
Costin

--
You received this message because you are subscribed to a topic in the Google Groups
"elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/__topic/elasticsearch/__Md6pwsOI2Dw/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@__googlegroups.com
.
To view this discussion on the web visit

https://groups.google.com/d/__msgid/elasticsearch/551BB843.__3040206%40gmail.com

.
For more options, visit https://groups.google.com/d/__optout
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
elasticsearch+unsubscr...@googlegroups.com
.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGgBrpwA5UF3VAyX%3D9tZMQbFxZ8BOREJyfEdiQ6G%2BSzXiqGyPg%40mail.gmail.com

How to get aggregations working in Elasticsearch Spark adapter ?

2015-04-01 Thread michele crudele


I have ES, Spark, and ES hadoop adapter installed on my laptop. I wrote a 
simple scala notebook to test ES adapter.
Everything was fine until I started thinking at more sophisticated 
features. This is the snippet that drives me crazy:

%AddJar 
file:///tools/elasticsearch-hadoop-2.1.0.Beta3/dist/elasticsearch-hadoop-2.1.0.BUILD-SNAPSHOT.jar
%AddJar 
file:///tools/elasticsearch-hadoop-2.1.0.Beta3/dist/elasticsearch-spark_2.10-2.1.0.BUILD-SNAPSHOT.jar

import org.elasticsearch.spark.rdd._

val q2 = """{
|"query" : { "term": { "appName": "console" } },
|"aggregations": {
|  "unusual": {
|"significant_terms": {"field": "pathname"}
|  }
|}
|}""".stripMargin

val res = sc.esRDD("logs/app", q2);

println("Matches: " + res.count())


When I run the code I get this exception:

Name: org.apache.spark.SparkException
Message: Job aborted due to stage failure: Task 2 in stage 15.0 failed 1 times, 
most recent failure: Lost task 2.0 in stage 15.0 (TID 58, localhost): 
org.apache.spark.util.TaskCompletionListenerException: 
SearchPhaseExecutionException[Failed to execute phase [init_scan], all shards 
failed; shardFailures {[N1R-UlgOQCGXCFCtbJ3sBQ][logrecords][2]: 
ElasticsearchIllegalArgumentException[aggregations are not supported with 
search_type=scan]}]
at 
org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:76)
at org.apache.spark.scheduler.Task.run(Task.scala:58)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:200)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)


"aggregations are not supported with search_type=scan", which is fine.
The question is: how do I set search_type to the right value (e.g. count) in 
the sc.esRDD() call? 
I tried several places in the q2 json with no success and I was not able to 
find an answer through
the documentation. I would appreciate any help.

However, I see a possible inconsistency with the behaviour of the ES API used 
directly via cURL.
The command with the same query above, and without any setting about 
search_type works correctly:

curl 'localhost:9200/logs/app/_search?pretty' -d'{"query" : { "term": { 
"appName": "console" } },
"aggregations": { "unusual": { "significant_terms": {"field": "pathname"} }}}'

returns hits:{} and aggregations:{}. Why the Spark integration does not work 
the same ?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d044d380-a4b2-4d22-8990-60f318f7601a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Error Could not contact Elasticsearch at http://:9200. Please ensure that Elasticsearch is reachable from your system. 1 alert(s)

2015-04-01 Thread Sergey Zemlyanoy

ES - 1.4.2
Kibana - 3.1.0

Please note that it worked earlier for us

Regards
Sergey

On Tuesday, March 31, 2015 at 5:37:20 AM UTC+3, Mark Walkom wrote:
>
> What versions of ES and KB are you running?
>
> On 30 March 2015 at 23:17, Sergey Zemlyanoy  > wrote:
>
>> Hi,
>>
>> It's global problem in our environment. We cannot open Kibana UI all 
>> browsers but Internet Explorer. 
>> What might be misconfigured? ELK stack in general looks sane.
>>
>> Regards
>> Sergey
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/f0b39cd5-5ddf-42f9-b936-1dab13d0e0ac%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/312bef97-9378-49f2-b341-aa1d0b2b60f3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to make unstructured data available in Hadoop for analysis with Elasticsearch?

2015-04-01 Thread 'BEN SALEM Omar' via elasticsearch

Hi Costin,

I've already done that and here I am using Kibana to dashboards and so on(I
have a question for you I'll ask a little bit later)

But, my question now is how to make UNstructured data available for such
analytics?
Let's suppose I have images or textfile that I want to analyse? How make it
available in a structured way in hive to do the required steps untill
loading it into my ES cluster and perform my analytics?

The second question is , I have created some Dashboards with Kibana; now
what I want to do, is giving my dashboards to the client I'm working
with/for to to do his analytics without letting him see what's going on the
ES cluster.
What's the easiest way to do this?
Thanks,
ᐧ

On Wed, Apr 1, 2015 at 11:20 AM, Costin Leau  wrote:

> If your data is in Hadoop, you would index it in Elasticsearch (through
> the es-hadoop connector) and then run your queries against it (either from
> Hadoop through the same connector) or through other tools (like Kibana)
> outside Hadoop.
>
> There are plenty of blog posts, documentations, presentations and videos
> on the topic so I recommend spending sometime time with them to better
> understand what options suite your use case.
>
> On 4/1/15 12:12 PM, BEN SALEM Omar wrote:
>
>> Hi everyone,
>> I'm wondering, it possible to make analytics on unstructured data with
>> Elasticsearch?
>> If so? how?
>> Thanks,
>> Omar,
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to
>> elasticsearch+unsubscr...@googlegroups.com > unsubscr...@googlegroups.com>.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/08e7617c-
>> 9e72-40a6-a780-7f632fee6d47%40googlegroups.com
>> > 9e72-40a6-a780-7f632fee6d47%40googlegroups.com?utm_medium=
>> email&utm_source=footer>.
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
> Costin
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/elasticsearch/Md6pwsOI2Dw/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/elasticsearch/551BB843.3040206%40gmail.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGgBrpwA5UF3VAyX%3D9tZMQbFxZ8BOREJyfEdiQ6G%2BSzXiqGyPg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

KIbana doesn't show log events that have resulted in _grokparsefailure

2015-04-01 Thread Pradeep Reddy

I have AWS ELB logs being indexed with production-elb-.ww, S3 access 
logs being indexed with production-s3-.ww
Default index pattern is production-s3*
I also have an index pattern for production-elb* 

When I select production-elb* and make a search for "_grokparsefailure" , 
it doesn't return any results.

If I create an index pattern production-* and search for "_grokparsefailure" 
, it does return the failed log events. I think both should return these 
log events.
One of these failed events is 

{
  "_index": "production-elb-2015.04",
  "_type": "elb-access-logs",
  "_id": "AUxzyxEQrRg0e6fTsE_9",
  "_score": null,
  "_source": {
"message": "2015-04-01T06:23:01.407429Z xxx xxx:yyy - -1 -1 -1 504 0 0 0 
\"GET https://xxx:443/ HTTP/1.1\"\n",
"@version": "1",
"@timestamp": "2015-04-01T07:03:49.339Z",
"type": "elb-access-logs",
"tags": [
  "_grokparsefailure"
]
  },
  "fields": {
"@timestamp": [
  1427871829339
]
  },
  "highlight": {
"message": [
  "2015-04-01T06:23:01.407429Z xxx xxx:yyy - -1 -1 -1 
@kibana-highlighted-field@504@/kibana-highlighted-field@ 0 0 0 
\"@kibana-highlighted-field@GET@/kibana-highlighted-field@ https://xxx:443/ 
HTTP/1.1\"\n"
],
"tags": [
  "@kibana-highlighted-field@_grokparsefailure@/kibana-highlighted-field@"
]
  },
  "sort": [
1427871829339
  ]
}

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d392523e-7585-425f-b8f6-c2a291c6be49%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elastichsearch CDH Hadoop Integration

2015-04-01 Thread Ravi sai kumar

Thanks Costin Leau.
My data will be in terabytes local mode doesnt suits my requirement may be. 
If i set the hadoop path,
Are there any changes we have to do for gateway.type?

existing property is 
gateway.type: local


On Wednesday, April 1, 2015 at 5:14:16 PM UTC+8, Costin Leau wrote:
>
> For best results and performance, point Elasticsearch to your local 
> storage. 
> It's just like any other service (MySQL, Postgres, etc..) 
>
> On 4/1/15 11:48 AM, Ravi sai kumar wrote: 
> > I have the problem with elastichsearch integrating with hadoop. 
> > I am new to elasticsearch. I have installed on CDH Single node cluster. 
> Please someone suggest me what should we keep at 
> > path.data: /path to data 
> > 
> > that means, Indexed data location should be hdfs or local directory? 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups "elasticsearch" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to 
> > elasticsearc...@googlegroups.com   elasticsearch+unsubscr...@googlegroups.com >. 
> > To view this discussion on the web visit 
> > 
> https://groups.google.com/d/msgid/elasticsearch/b750ba78-360f-47af-8d8b-cb79e9e346f9%40googlegroups.com
>  
> > <
> https://groups.google.com/d/msgid/elasticsearch/b750ba78-360f-47af-8d8b-cb79e9e346f9%40googlegroups.com?utm_medium=email&utm_source=footer>.
>  
>
> > For more options, visit https://groups.google.com/d/optout. 
>
> -- 
> Costin 
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/05d4ef8a-e1a4-47ef-ac26-7b4aada7cfac%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to make unstructured data available in Hadoop for analysis with Elasticsearch?

If your data is in Hadoop, you would index it in Elasticsearch (through the es-hadoop connector) and then run your 
queries against it (either from Hadoop through the same connector) or through other tools (like Kibana) outside Hadoop.


There are plenty of blog posts, documentations, presentations and videos on the topic so I recommend spending sometime 
time with them to better understand what options suite your use case.


On 4/1/15 12:12 PM, BEN SALEM Omar wrote:

Hi everyone,
I'm wondering, it possible to make analytics on unstructured data with 
Elasticsearch?
If so? how?
Thanks,
Omar,

--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to
elasticsearch+unsubscr...@googlegroups.com 
.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/08e7617c-9e72-40a6-a780-7f632fee6d47%40googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.

--
Costin

--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/551BB843.3040206%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Try to make es-hadoop run

See the configuration section of the page, in particular the es.version property. It defaults to 1.4.0 but you can
change it to

1.5.0 or any other version.

On 4/1/15 12:10 PM, stéphane Verdy wrote:

Hi Costin,

Thank you for your reply. Visibly I was not the right way although
following step by step _ online's documentation
which you have given me the link!

Have you the right link for downloading the latest version of Elastic on YARN?
I can't find it by myself sorry

Regards,
Stéphane

2015-03-31 17:04 GMT+02:00 Costin Leau mailto:costin.l...@gmail.com>>:

Debugging Hadoop is ... tricky. Most likely the parameter you are passing
(es.version) is incorrect - there's no
such Elasticsearch version.
2.1.0.Beta3 is the connector, es-hadoop version. es.version indicates the
version of Elasticsearch itself.

My advice is to first start with basic steps and defaults as described in
the docs [1].

As for the debug part itself, you have to look into your node/cluster logs,
in particular the yarn/node ones to see
the underlying container exception.
Whatever message/exception is thrown by the underlying container, it does
not bubble up (it is always wrapped by
YARN itself) and thus one has to manually
track it down...

[1]
http://www.elastic.co/guide/__en/elasticsearch/hadoop/2.1.__Beta/ey-usage.html

On 3/31/15 5:50 PM, stéphane Verdy wrote:

Hi all,

HDFS 2.6.0.2.2.0.0 Apache Hadoop Distributed File System
Es-Hadoop : 2.1.0.Beta3
OS : Centos 6

When i start elasticsearch (hadoop jar
elasticsearch-yarn-2.1.0.__Beta3.jar -start es.version=2.1.0.Beta3), the
process end FINISHED/FAILED and the only log i found is :

2015-03-31 13:29:57,792 INFO nodemanager.ContainerExecutor
(ContainerExecutor.java:__logOutput(283)) - at
java.lang.Thread.run(Thread.__java:744)
2015-03-31 13:29:57,793 WARN launcher.ContainerLaunch
(ContainerLaunch.java:call(__338)) - Container exited with a
non-zero exit code 127

No ERROR log only WARN...
Can'i set the logger in DEBUG mode (how) ?
Are there any other log files ?

Thank
Stephane

>.
To view this discussion on the web visit

https://groups.google.com/d/__msgid/elasticsearch/82bea02f-__7356-4bab-8895-00b26842352a%__40googlegroups.com

>.
For more options, visit https://groups.google.com/d/__optout
.

--
Costin

--
You received this message because you are subscribed to a topic in the Google Groups
"elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/__topic/elasticsearch/__HihYoW8snuc/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@__googlegroups.com
.
To view this discussion on the web visit

https://groups.google.com/d/__msgid/elasticsearch/551AB786.__1040107%40gmail.com

For more options, visit https://groups.google.com/d/__optout
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
elasticsearch+unsubscr...@googlegroups.com
.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAMrSTxdL0KvYYC%2BWx6ZXAf3SygJHjXpbpYBEdKcbC82g68-RBA%40mail.gmail.com
.
For more options, visit https://groups.google.com/d/optout.

--
Costin

Re: Elastichsearch CDH Hadoop Integration


For best results and performance, point Elasticsearch to your local storage.
It's just like any other service (MySQL, Postgres, etc..)

On 4/1/15 11:48 AM, Ravi sai kumar wrote:

I have the problem with elastichsearch integrating with hadoop.
I am new to elasticsearch. I have installed on CDH Single node cluster. Please 
someone suggest me what should we keep at
path.data: /path to data

that means, Indexed data location should be hdfs or local directory?

--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to
elasticsearch+unsubscr...@googlegroups.com 
.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b750ba78-360f-47af-8d8b-cb79e9e346f9%40googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.


--
Costin

--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/551BB6DB.7060203%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

How to make unstructured data available in Hadoop for analysis with Elasticsearch?