date:20150402

You can do data transformation on the fly, yes.

Language detection can't be done in LS that I know of, but you can
definitely trim things.

On 3 April 2015 at 13:16, Employ  wrote:

> Thank you for the reply. I've seen that mentioned but does it have the
> capability to modify the XML content before it is imported? For example,
> adding the ability to do language detection and trimming via custom scripts?
>
> On 2 Apr 2015, at 19:44, Mark Walkom  wrote:
>
> Logstash can handle XML, it has a filter specifically for it -
> http://www.elastic.co/guide/en/logstash/current/plugins-filters-xml.html
>
> On 3 April 2015 at 09:33, James  wrote:
>
>> Hi,
>>
>> Currently I am using scrapy to parse an XML file from an ftp server into
>> elasticsearch. It works but seems quite a heavy weight solution and it uses
>> a lot of memory too.
>>
>> I am wondering if I am better off writing a plugin for ES instead.
>>
>> I have some questions:
>>
>> A) It seems writing it in Python (since I'm a python guy) as a push
>> plugin rather than a pull river makes sense, unless anyone has a reason why
>> pull is better?
>>
>> B) For simple importing (and slight modification such as trimming,
>> language check etc) is it likely that an ES plugin is likely going to be a
>> better solution to importing fairly large XML files or should I just leave
>> scrapy to do it as it is doing at the moment?
>>
>> Any help and advice would be appreciated as I start on this journey.
>>
>> James
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/610e7f9b-3d23-44a9-b8f3-07deb262dd54%40googlegroups.com
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/L9uzIGfT7Gs/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8TLso3YjNLpqHoR5r87nr6Li2Ng53AjHwwNzE1j9FJeA%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/AE7A0FB1-0DE9-4BBF-BEE1-7A29964204E5%40employ.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X92a7G536TNgArbzdvC9P%2B0gKMS_5jMBxT9ZBVDJ9PMMg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch Scan and Scroll

2015-04-02 Thread Yogesh

Yes. If you are using the Python API, have a look at the scan helper: 
http://elasticsearch-py.readthedocs.org/en/master/helpers.html#elasticsearch.helpers.scan

Yogesh

On Thursday, April 2, 2015 at 3:56:48 PM UTC+5:30, lakshay joshi wrote:
>
> Hi All,
>
> My question is related to Scan and Scroll API.
> Can we include SCAN and SCROLL with in single elasticsearch query template 
> to get all the results.
>
> Thanks & Regards
> Lakshay Joshi
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/03ae59a0-f26f-4213-8e26-8b5974e1d648%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Case Insensitive Sort

2015-04-02 Thread Sridharkalaibala

Thanks you very much... Its working... 



--
View this message in context: 
http://elasticsearch-users.115913.n3.nabble.com/Case-Insensitive-Sort-tp4021310p4073006.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1427904326005-4073006.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.

Re: Getting XML into ES efficiently

2015-04-02 Thread Employ

Thank you for the reply. I've seen that mentioned but does it have the 
capability to modify the XML content before it is imported? For example, adding 
the ability to do language detection and trimming via custom scripts?

> On 2 Apr 2015, at 19:44, Mark Walkom  wrote:
> 
> Logstash can handle XML, it has a filter specifically for it - 
> http://www.elastic.co/guide/en/logstash/current/plugins-filters-xml.html
> 
>> On 3 April 2015 at 09:33, James  wrote:
>> Hi,
>> 
>> Currently I am using scrapy to parse an XML file from an ftp server into 
>> elasticsearch. It works but seems quite a heavy weight solution and it uses 
>> a lot of memory too.
>> 
>> I am wondering if I am better off writing a plugin for ES instead.
>> 
>> I have some questions:
>> 
>> A) It seems writing it in Python (since I'm a python guy) as a push plugin 
>> rather than a pull river makes sense, unless anyone has a reason why pull is 
>> better?
>> 
>> B) For simple importing (and slight modification such as trimming, language 
>> check etc) is it likely that an ES plugin is likely going to be a better 
>> solution to importing fairly large XML files or should I just leave scrapy 
>> to do it as it is doing at the moment?
>> 
>> Any help and advice would be appreciated as I start on this journey.
>> 
>> James
>> 
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/610e7f9b-3d23-44a9-b8f3-07deb262dd54%40googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
> 
> -- 
> You received this message because you are subscribed to a topic in the Google 
> Groups "elasticsearch" group.
> To unsubscribe from this topic, visit 
> https://groups.google.com/d/topic/elasticsearch/L9uzIGfT7Gs/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to 
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8TLso3YjNLpqHoR5r87nr6Li2Ng53AjHwwNzE1j9FJeA%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/AE7A0FB1-0DE9-4BBF-BEE1-7A29964204E5%40employ.com.
For more options, visit https://groups.google.com/d/optout.

Re: elastic search query syntax

2015-04-02 Thread buddarapu nagaraju

also please let me know the syntax for below query scenario

how to find the documents with  a phrase starts with "united" and ends with
"states" conatain the term "new york" in between

Regards
Nagaraju
908 517 6981

On Thu, Apr 2, 2015 at 8:38 PM, bvnrwork  wrote:

> Can some one help me to write queries for below scenarios ?
>
> how to find documents that has united and states but not with in 30 words
> ?(united AND states NOT WITHIN 30 WORDS)?
>
> how to find the documents that has united before america ?(united BEFORE
> america)
>
> how to find the documents that has united followed by america?(united
> FOLLOWEDBY america )
>
> how to find the documents that war and peace near novel or book?(war AND
> peace) NEAR (novel OR book)
>
>
> Regards,
>
> Nagaraju
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/rqDipmvYjyU/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/074d907d-1e87-4c57-9933-227e407fd0cb%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAFtuXXKNAGkQdQhvDVcvgYZEyxurbCFyds%3DMWmCk2c07GPLr5w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

elastic search query syntax

2015-04-02 Thread bvnrwork

Can some one help me to write queries for below scenarios ?

how to find documents that has united and states but not with in 30 words 
?(united 
AND states NOT WITHIN 30 WORDS)?

how to find the documents that has united before america ?(united BEFORE 
america)

how to find the documents that has united followed by america?(united 
FOLLOWEDBY america ) 

how to find the documents that war and peace near novel or book?(war AND 
peace) NEAR (novel OR book)


Regards,

Nagaraju

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/074d907d-1e87-4c57-9933-227e407fd0cb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Is there a way to do scan with limit?

2015-04-02 Thread Chen Wang

I want to for example, fast get 1m out of 5m records.
I am currently using:

SearchResponse scrollResp = this.client
 .prepareSearch(esQuery.indices) 

 .addFields(esQuery.fields) 

 .setSearchType(SearchType.SCAN) 

 .setScroll( 

 TimeValue 

 .timeValueSeconds(this.scrollTimeInSeconds)) 

 .setQuery(esQuery.query) 

 .setSize(this.queryBatchSizePershard).execute().actionGet();


but the setSize defines how many records return per shard in one scroll. Is 
there a way to define a limit? or I have to control the limit in my code.

Chen

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5fb4751f-7080-423c-9acd-bfd56abfd844%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to enable Garbage Collector logging in an elastic search nodes..

We don't recommend G1GC as it can lead to data loss and corruption in the
underlying lucene engine.

On 2 April 2015 at 16:49, Anil Karaka  wrote:

> Ok, we also had to edit that file to change the default garbage collector
> options and are now using G1GC. So I thought I will just change the if
> condition to enable gc logs..
>
> Any way, this is not the main problem, I'm trying to do all these things
> to debug the main issue of my nodes leaving the cluster randomly.
> https://groups.google.com/forum/#!topic/elasticsearch/E-aGhovVTPI here
> are more details. Any help would be appreciated.
>
> Yesterday night I increased the zen.ping.timeout from 3s to 10s of all my
> machines and restarted them one by one, and yet to see if its any
> improvement.
> Will updating elasticsearch in my nodes fix my issues? Currently my nodes
> are using ES-1.4.1.
>
> Thanks.
>
> On Thursday, April 2, 2015 at 3:02:31 AM UTC+5:30, Mark Walkom wrote:
>>
>> It's not recommended to alter the .in.sh, this file can change between
>> releases and your alterations can either be lost or can cause other
>> problems.
>>
>> I'd suggest you do what Gavin mentioned.
>>
>> On 1 April 2015 at 18:08, Anil Karaka  wrote:
>>
>>> Thanks,
>>>
>>> I just kept those GC logging options outside the if block.. So that I
>>> don't have to set that variable, and enabled GC logging by default.
>>>
>>> this is in the file /usr/share/elasticsearch/bin/elasticsearch.in.sh
>>>
>>> # GC logging options
>>> if [ "x$ES_USE_GC_LOGGING" != "x" ]; then
>>>   JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails"
>>>   JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCTimeStamps"
>>>   JAVA_OPTS="$JAVA_OPTS -XX:+PrintClassHistogram"
>>>   JAVA_OPTS="$JAVA_OPTS -XX:+PrintTenuringDistribution"
>>>   JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCApplicationStoppedTime"
>>>   JAVA_OPTS="$JAVA_OPTS -Xloggc:/var/log/elasticsearch/gc.log"
>>> fi
>>> # Enable gc logging by default
>>> JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails"
>>> JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCTimeStamps"
>>> JAVA_OPTS="$JAVA_OPTS -XX:+PrintClassHistogram"
>>> JAVA_OPTS="$JAVA_OPTS -XX:+PrintTenuringDistribution"
>>> JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCApplicationStoppedTime"
>>> JAVA_OPTS="$JAVA_OPTS -Xloggc:/var/log/elasticsearch/gc.log"
>>>
>>>
>>> On Saturday, March 14, 2015 at 7:18:29 AM UTC+5:30, Gavin Seng wrote:


 Anil, you can try this:

 # /etc/sysconfig/elasticsearch

 export ES_USE_GC_LOGGING=1
 ES_JAVA_OPTS="-verbose:gc -XX:+PrintGCDateStamps
 -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M
 -Xloggc:"

 Gavin

 On Wednesday, March 4, 2015 at 9:22:50 AM UTC-5, Anil Karaka wrote:
>
> Can you please tell me what exactly do I need to add in
> /etc/default/elasticsearch?
>
> What should I set ES_USE_GC_LOGGING to?
>
> I added below line in my dev ES node, and I didn't see the gc.log file
> in my log folder.. I want to be sure what to do before changing anything 
> in
> my production cluster..
>
> ES_USE_GC_LOGGING=true
>
> And what are the GC logging options in my elasticsearch.yml do? where
> do those logs appear?
>
> Thank you.
>
> On Wednesday, March 4, 2015 at 3:11:26 AM UTC+5:30, Mark Walkom wrote:
>>
>> You can set JAVA_OPTS in /etc/default/elasticsearch or
>> /etc/sysconfig/elasticsearch u9nder ES_JAVA_OPTS
>>
>> On 3 March 2015 at 23:11, Anil Karaka  wrote:
>>
>>> Some of the nodes are leaving the cluster from time to time..
>>>
>>> I un commented the GC Logging options in elasticsearch.yml file as
>>> shown and restarted my node.
>>>
>>> ## GC Logging
>>> 
>>>
>>> monitor.jvm.gc.young.warn: 1000ms
>>> monitor.jvm.gc.young.info: 700ms
>>> monitor.jvm.gc.young.debug: 400ms
>>>
>>> monitor.jvm.gc.old.warn: 10s
>>> monitor.jvm.gc.old.info: 5s
>>> monitor.jvm.gc.old.debug: 2s
>>>
>>>
>>> yet I'm not able to see any of the GC logging..
>>>
>>> I came across this blog post http://tech.taskrabbit.com/blo
>>> g/2014/07/18/elasticsearch-in-production/
>>>
>>> `ElasticSearch will note in its normal logs if any GC is happening
>>> which effects a slow query, but we wanted more detail. Another logging
>>> option ElasticSearch has is ES_USE_GC_LOGGING which sets some JVM
>>> variables:
>>>
>>> # from elasticsearch.in.sh.erbif [ "x$ES_USE_GC_LOGGING" != "x" ]; then
>>>   JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails"
>>>   JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCTimeStamps"
>>>   JAVA_OPTS="$JAVA_OPTS -XX:+PrintClassHistogram"
>>>   JAVA_OPTS="$JAVA_OPTS -XX:+PrintTenuringDistribution"
>>>   JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCApplicationStoppedTime"
>>>   JAVA_OPTS="$JAVA_OPTS -Xloggc:/var/log/elasticsearch/gc.log"fi
>>>
>>> `
>>> Where do I set the ES_USE_GC_LOGGI

Re: Getting XML into ES efficiently

Logstash can handle XML, it has a filter specifically for it -
http://www.elastic.co/guide/en/logstash/current/plugins-filters-xml.html

On 3 April 2015 at 09:33, James  wrote:

> Hi,
>
> Currently I am using scrapy to parse an XML file from an ftp server into
> elasticsearch. It works but seems quite a heavy weight solution and it uses
> a lot of memory too.
>
> I am wondering if I am better off writing a plugin for ES instead.
>
> I have some questions:
>
> A) It seems writing it in Python (since I'm a python guy) as a push plugin
> rather than a pull river makes sense, unless anyone has a reason why pull
> is better?
>
> B) For simple importing (and slight modification such as trimming,
> language check etc) is it likely that an ES plugin is likely going to be a
> better solution to importing fairly large XML files or should I just leave
> scrapy to do it as it is doing at the moment?
>
> Any help and advice would be appreciated as I start on this journey.
>
> James
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/610e7f9b-3d23-44a9-b8f3-07deb262dd54%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8TLso3YjNLpqHoR5r87nr6Li2Ng53AjHwwNzE1j9FJeA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Getting XML into ES efficiently

2015-04-02 Thread James

Hi,

Currently I am using scrapy to parse an XML file from an ftp server into 
elasticsearch. It works but seems quite a heavy weight solution and it uses a 
lot of memory too.

I am wondering if I am better off writing a plugin for ES instead. 

I have some questions:

A) It seems writing it in Python (since I'm a python guy) as a push plugin 
rather than a pull river makes sense, unless anyone has a reason why pull is 
better?

B) For simple importing (and slight modification such as trimming, language 
check etc) is it likely that an ES plugin is likely going to be a better 
solution to importing fairly large XML files or should I just leave scrapy to 
do it as it is doing at the moment?

Any help and advice would be appreciated as I start on this journey.

James

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/610e7f9b-3d23-44a9-b8f3-07deb262dd54%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Set index.query.parse.allow_unmapped_fields = false for a single index

You can use index.mapper.dynamic: false

See
http://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-mapper.html#_dynamic_mappings



On 3 April 2015 at 09:14, Todd Nine  wrote:

> Figured this out (derp).  So my next question, is how can I disable
> dynamic mappings so that they don't occur?
>
> Thanks,
> Todd
>
> On Thursday, April 2, 2015 at 3:55:07 PM UTC-6, Todd Nine wrote:
>>
>> Hey guys,
>>   We're running ES as a core service, so I can't guarantee that my
>> application will be the only one using the cluster.  Is it possible to
>> disallow unmapped fields explicitly for an index without modifying the
>> global setting?
>>
>> Thanks,
>> Todd
>>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/29f0c5e3-068d-4e19-9a3b-6cd38ec60e08%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9pcUgD0oSEPFS-UWAOjjX8JqKbmVfaa0DC0PcZKtspvw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Set index.query.parse.allow_unmapped_fields = false for a single index

2015-04-02 Thread Todd Nine

Figured this out (derp).  So my next question, is how can I disable dynamic 
mappings so that they don't occur?

Thanks,
Todd

On Thursday, April 2, 2015 at 3:55:07 PM UTC-6, Todd Nine wrote:
>
> Hey guys,
>   We're running ES as a core service, so I can't guarantee that my 
> application will be the only one using the cluster.  Is it possible to 
> disallow unmapped fields explicitly for an index without modifying the 
> global setting?
>
> Thanks,
> Todd
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/29f0c5e3-068d-4e19-9a3b-6cd38ec60e08%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

_bulk API: a field can't be removed unless deleting the document first and then upsert

2015-04-02 Thread Thong Bui

Hi all,

Here is the scenario:

1) I first populate the data record in Elasticsearch using _bulk API:
- here is temp.json look like:
{ update : {"_id" : "Tra." }}
{ doc : {"field1":"Test 1",  "field2" : "Test 2" }, doc_as_upsert : true }

- I then load it using _bulk api:
curl -s -XPOST http://localhost:9200/thong_test/test/_bulk --data-binary 
@temp.json

2) I then want to remove data in field2, but changing temp.json as below 
still leave field2 behind:

{ update : {"_id" : "Tra." }}
{ doc : {"field1":"Test 1",  }, doc_as_upsert : true }

If I delete it first, then it will work:

{ delete: {"_id" : "Tra." }}
{ update : {"_id" : "Tra." }}
{ doc : {"field1":"Test 1",  "field2" : "Test 2" }, doc_as_upsert : true }

However, is there a better to remove a field in _bulk without delete and 
insert the document?

Thanks so much for your help!

Thong Bui



-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/6e2640e0-2ad9-45fe-8d2f-9501388f5d05%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Is it possible to rename a field in ES?

You can create an alias with a filter on a field and then use that, but you
cannot alias an individual field.

On 2 April 2015 at 16:25, Johnny Wang  wrote:

> Or can I add an alias for 'foo'?
>
> On Saturday, September 17, 2011 at 5:59:11 AM UTC+8, kimchy wrote:
>>
>> Yes, once a field is set, there is no way to rename it.
>>
>> On Fri, Sep 16, 2011 at 10:16 PM, Dan Young  wrote:
>>
>>> I believe you'll need to reindex.
>>>
>>>
>>>
>>> On Fri, Sep 16, 2011 at 1:13 PM, datadev  wrote:
>>>
 Suppose I create an index X with type Y and add 100,000 documents
 containing a field 'foo'.

 Is there a command I can use to atomically rename the field name from
 'foo' to 'bar', or would I have to reindex all 100,000 documents with the
 new field name?

 Thanks.

>>>
>>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/b8eb7181-bf2c-4367-bb7c-dc3f1244765e%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9oo_g2T%2BnFw5ZSP2H4U6c1-6E3oUNx5b8h4AHYDvngKg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Set index.query.parse.allow_unmapped_fields = false for a single index

2015-04-02 Thread Todd Nine

Hey guys,
  We're running ES as a core service, so I can't guarantee that my 
application will be the only one using the cluster.  Is it possible to 
disallow unmapped fields explicitly for an index without modifying the 
global setting?

Thanks,
Todd

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/19b5a2f9-88f4-4100-809d-3e258e24f6af%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Security Suggestion In Elasticsearch

You don't need to use both, no. I've heard of people using
elasticsearch-jetty successfully, but it appears to have been unmaintained
for a while now.

And yes, Shield is a commercial plugin, you can try it free for 30 days
though.

On 2 April 2015 at 16:09, Shohedul Hasan  wrote:

> Do i have to use both of iptable and nginx.  Another thing what is about
> elasticsearch-jetty, will it be better than nginx?  I didn't know that
> shield is not free.
>
> On Tuesday, March 31, 2015 at 1:43:25 PM UTC+6, Mark Walkom wrote:
>>
>> Shield would be the best.
>> But you can also use iptables and nginx to secure and proxy things.
>>
>> On 31 March 2015 at 17:45, Shohedul Hasan  wrote:
>>
>>> Hi,
>>> I am trying to deploy my ES server in Digital ocean. But Digital ocean
>>>  had some hacker attack as i didn't add any security to ES, So they blocked
>>> my droplet and gave me warning. So i tried to find out how can i secure my
>>> ES server. I have found out that Shield , using proxy or some other ways
>>> can be  solution. But i am confused.  What is the best option (to secure
>>> ES) i have right now.
>>>
>>> 1) I have one ES server (i may add more in future)
>>> 2) One server that run the java code. From this server i do all ES
>>> operation.
>>> 3) There are some more servers like postgresql, gearman but i think
>>> these are not relevant here.
>>>
>>> Thanks in advance.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/elasticsearch/4aa2f9f8-b4b7-4ce3-8ff3-66c3304d64a5%
>>> 40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/4f8191d4-750e-4574-b3fb-f6375b3f4b19%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-QmpZagpGE-jJuiwXJM0KWVY6vBBHuZe5rCvb5zMZsAQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Expand sharding on exist index

You will have to reindex from an external source then.

On 2 April 2015 at 22:43, Grigory Rubstein  wrote:

> ThaT
> Thank you for the response ,
> regarding second option, i want to reindex current index, as the
> performance is already very slow and we can leave it as is.
> the problem is that we have types that not stored ( "_source" : {
> "enabled" : false } ) how can we reindex such doc types without retrieving
> them ? is there any special api for this purpose ?
>
>
>
> On Thursday, April 2, 2015 at 11:31:23 AM UTC+3, Grigory Rubstein wrote:
>>
>> Hi
>>
>> Is its possible to change number of shards on exist index, where part of
>> the types does not stored fields ?
>>
>> This message may contain confidential and/or privileged information.
>> If you are not the addressee or authorized to receive this on behalf of
>> the addressee you must not use, copy, disclose or take action based on this
>> message or any information herein.
>> If you have received this message in error, please advise the sender
>> immediately by reply email and delete this message. Thank you.
>>
>
> This message may contain confidential and/or privileged information.
> If you are not the addressee or authorized to receive this on behalf of
> the addressee you must not use, copy, disclose or take action based on this
> message or any information herein.
> If you have received this message in error, please advise the sender
> immediately by reply email and delete this message. Thank you.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/475226eb-fb9d-4f8a-ae60-f71ac58670ea%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8OJN5vQRmyLkfXxxHqC1RkfkFVP5O5FeHnMxXDfr0nWA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: ElasticSearch Multi Data Center replication

If you don't want to snapshot+restore then use your code to write to each
cluster.

On 2 April 2015 at 23:06, Vladi Feigin  wrote:

> Thank you Itamar,
> Can you describe the process with the more details?
> Is it something like that:
>
> 1. Extract a snapshot in cluster_source with snapshot API
> 2. Copy it to target cluster
> 3. Restore it in the target cluster with restore API
>
> All this wrap in the two daemon jobs , one for snapshot (running in the
> source cluster) and other for restoring running in the target cluster?
> Thank you,
> Vladi
>
>
> On Thu, Apr 2, 2015 at 2:58 PM, Itamar Syn-Hershko 
> wrote:
>
>> Use the snapshot / restore API, unless you care for latency then look at
>> implementing a daemon tool to do that. You should try to keep them decoupled
>>
>> --
>>
>> Itamar Syn-Hershko
>> http://code972.com | @synhershko 
>> Freelance Developer & Consultant
>> Lucene.NET committer and PMC member
>>
>> On Thu, Apr 2, 2015 at 2:48 PM, Vladi Feigin 
>> wrote:
>>
>>> Hi,
>>>
>>> Is there a solution for this ? Could you recommend some production
>>> proven?
>>> Thank you
>>> Vladi
>>>
>>> This message may contain confidential and/or privileged information.
>>> If you are not the addressee or authorized to receive this on behalf of
>>> the addressee you must not use, copy, disclose or take action based on this
>>> message or any information herein.
>>> If you have received this message in error, please advise the sender
>>> immediately by reply email and delete this message. Thank you.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearch+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elasticsearch/bb6ef682-5a94-4f2e-9cfb-345a9c31ffa7%40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "elasticsearch" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/elasticsearch/iRMQb8vVG_w/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtAMWcLLtJP0WTgFkofBBa04wW%2Buypx456BnQYLmVwmvw%40mail.gmail.com
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
> This message may contain confidential and/or privileged information.
> If you are not the addressee or authorized to receive this on behalf of
> the addressee you must not use, copy, disclose or take action based on this
> message or any information herein.
> If you have received this message in error, please advise the sender
> immediately by reply email and delete this message. Thank you.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CACvWdiqR%2B9trJZ6gv2fbf0FiyEsL_C%2BazrE7D4V0L7aw3T%3D_Sg%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8QtEo69QOh2JvpTMawmTK%2Bp%3DkaMqZxGjTP7wRAoKfH9g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: UNASSIGNED indexes

Take a look in your ES logs, it should have something of use.

You can also try dropping the replicas to 0 for the indices that are having
these problems, then readd them one index at a time.

On 2 April 2015 at 23:42, Darius  wrote:

> Hello,
>
> Quick description of the problem: I had 1 elasticsearch server cluster
> with 0 replicas for all indexes, then added a second ES server to the
> cluster and then set that all existing indexes should have 1 replica
> (instead of the 0) .
> After doing this, in the _cat/shards page I could see that it created
> twice as many indexes/indexes(?) (sorry, im not that good with the ES
> terminology) and all the new ones were UNASSIGNED. They then started
> slowly INITIALIZING (2 at a time) and then go to STARTED state on the
> second server. So thats good, but the problem is that this was done like 2
> weeks ago and some of them are still UNASSIGNED or INITIALIZING. I've even
> noticed some of them going into the INITIALIZING state and then falling
> back to UNASSIGNED. How can I properly find the problem here and is there
> any way I can force the replicas to be allocated to the 2nd node WITHOUT
> any data loss? I found only one way to allocate, but that includes data
> loss, unfortunately.
>
> I don't have tons of data in there so I dont see why it should take so
> long. The largest index is ±500M in size, most of them are a few megs and
> the total number is around ±150-200.
>
> any help would be appreciated..
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/05cad3e5-78ce-44f7-9a30-6ca0eb817237%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8mYPddh6m1aGv60X1GVtjt%3Dq8VqzsTOAKmGj17TyYZAA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Support multiple data sources

Reading data from MySQL isn't supported and I doubt it will be, it's been
built purely to interact with Elasticsearch.

On 3 April 2015 at 05:03, Yosi Attias  wrote:

> Hi,
>
> I am currently using kibana and I really like it for dashboard - but most
> of my data is in the mssql or more than 3 elastic indexes.
> I would like to create one dashboard that gets data from mssql and from my
> other elastic indexes.
>
>
> Is this possible today? If not, kibana core committers I will be more than
> happy to create a PR for this, will you apply it?
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/e00336d1-26d5-4113-8eb8-b50d77ab26ee%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-EA6uVhf9kLsoCMRHWJPPrSS1AWvnmGN%3DGjyEQfta1zw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Alert notification with percolator

2015-04-02 Thread Roger de Cordova Farias

I have never used percolator, but afaik you have to call the percolator api
after you have the document indexed:

http://www.elastic.co/guide/en/elasticsearch/reference/current/search-percolate.html#_percolating_an_existing_document

2015-04-02 15:25 GMT-03:00 Lincoln Xiong :

> I try to use elasticsearch as a 2nd log output storage, to analyze some
> info in logs. In this case, an alert triggers will be very useful. I read
> through docs talking about percolator and I think this should be the way to
> make it out.But after some trying, I found that I don't really get how
> percolator works. It seems that I use REST API to index a document with a
> percolator already set up, it will return if that document match the
> percolator query or not. For my case, I use Logstash as input which of
> course don't have this kind of feedback. And a "count" appears to be
> accessible from REST that I can use to get this kind "feedback" from
> percolator but I find it no where.
>
> Could some one give me an idea about how I can achieve this kind of
> feature with Elasticsearch?
>
> I know I can have ways to trigger an alert in Logstash but for my case
> Logstash is a temporary tool to input the data, I could possible not use it
> in the future.
> I also notice that Graylog has kind of alert. When the input event match
> some keywords, the alarm will trigger. I guess it also use some percolator
> APIs but I wish to know how can I do this alone with Elasticsearch only.
>
> Thanks a lot.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/cf5da9d8-7000-4a0b-94af-3ce064feee90%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAJp2530%2BAkecJAbVnQ3E8X1_iWNqzAK%3DmbDkHgECAomZgnfdrg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Problem with ES- Shield

2015-04-02 Thread 'BEN SALEM Omar' via elasticsearch

Hey Cosin,
And when It comes to the marvel?
Can you see why It's not seeing my indices?
Thanks,

On Thu, Apr 2, 2015 at 9:46 PM, Costin Leau  wrote:

> The configuration example is correct - one can pass any configuration
> property there and it will be picked up as explained here [1]
>
> Additionally, to verify one can turn on logging [2] to see the connection
> is properly authenticated
>
> [1] http://www.elastic.co/guide/en/elasticsearch/hadoop/
> master/pig.html#pig-configuration
> [2] http://www.elastic.co/guide/en/elasticsearch/hadoop/
> master/logging.html
>
> On 4/2/15 9:21 PM, Jay Modi wrote:
>
>> Still not sure what's going on with Marvel, unfortunately. No idea if
>> this will help, but have you tried clearing your
>> browser cache or using a incognito window?
>>
>> I think something like this:
>>
>> TBLPROPERTIES('es.resource' = 't/t',
>> 'es.nodes'='192.168.238.130:9200 ',
>>  'es.net.http.auth.user'='omar',
>>  'es.net.http.auth.pass'='omar')
>>
>> I copy/pasted the wrong names earlier. I'm not a hadoop user but I
>> believe that should work based off of the following
>> documentation:
>> http://www.elastic.co/guide/en/shield/current/hadoop.html
>> http://www.elastic.co/guide/en/elasticsearch/hadoop/
>> current/hive.html#hive-configuration
>>
>> On Thursday, April 2, 2015 at 2:05:15 PM UTC-4, BEN SALEM Omar wrote:
>>
>> Jay,
>> when it comes to adding the .user and .password in the tblproperties
>> Do you know what exactly to do?
>> let's suppose that my admin esuser is omar and pwd omar
>> what should I do?
>> TBLPROPERTIES('es.resource' = 't/t',
>> 'es.nodes'='192.168.238.130:9200 ',
>>  and now what to write);
>> and is there any further config to do?
>> Thanks for your time
>> ᐧ
>>
>> On Thu, Apr 2, 2015 at 7:29 PM, BEN SALEM Omar > > wrote:
>>
>> Yes I had,
>> I used to manage my cluster with marvel perfectly and all the
>> indices used to show up.
>> But now, none of them seem to be found due to time span; and even
>> when I removed the shield this "porblem" occured.
>> I don't see how to "restore" the previous configuration.
>> Any idea?
>> ᐧ
>>
>> On Thu, Apr 2, 2015 at 7:14 PM, Jay Modi > > wrote:
>>
>> There should not be anything extra necessary to work with
>> Marvel. The browser prompted you for credentials
>> and you entered admin credentials, correct?
>>
>> On Thursday, April 2, 2015 at 12:39:03 PM UTC-4, BEN SALEM
>> Omar wrote:
>>
>> Or not, that wenti as I'd like to go :))
>> Here is the last remaining issue :
>>
>>
>> 
>> ᐧ
>>
>> On Thu, Apr 2, 2015 at 6:32 PM, BEN SALEM Omar <
>> omar.b...@esprit.tn> wrote:
>>
>> That would be a problem !
>> Last question (as for now :p ), how to change the
>> marvel timepicker? I can't see any indice of
>> mine(as I already told you)
>> And hey, thank you for your cooperation Jay
>> ᐧ
>>
>> On Thu, Apr 2, 2015 at 6:26 PM, Jay Modi <
>> j...@elastic.co> wrote:
>>
>> Hi Omar,
>>
>> I think you need to add "es.net.http.auth.user"
>> and "es.net.http.auth_password" as
>> additional TBLPROPERTIES. You will need to using
>> elasticsearch Hadoop 2.1 beta 3 for this to
>> work as far as I know.
>>
>> -Jay
>>
>> On Thursday, April 2, 2015 at 11:18:12 AM UTC-4,
>> BEN SALEM Omar wrote:
>>
>> Hi Jay,
>> Now that I have shield in both ES and Kibana,
>> and I want to load some data from my hadoop
>> cluster to ES.
>> I usually do this :
>> 
>> 
>> create external table es_word_count(word
>> string, count int)
>> STORED BY 'org.elasticsearch.hadoop.__
>> hive__.EsStorageHandler'
>> TBLPROPERTIES('es.resource' = 'test/test',
>> 'es.nodes'='192.168.238.130 <
>> http://192.168.238.130>:__92__00');
>> INSERT OVERWRITE TABLE es_word_count SELECT
>> s.word, s.count  FROM word_counts1 s;
>>
>>
>> what should I do or add to access to my ES
>> cluster as the esuser I've created?
>> Thanks
>> ᐧ
>>
>> On Thu, Apr 2, 2015 at 1:08 PM, Jay Modi <
>> j...@elastic.co> wrote:
>>
>> Hi Omar,
>>
>> Have you

Re: Setting publish_address for logstash

2015-04-02 Thread mailsijomathew

bind_host option did the trick!

On Thursday, April 2, 2015 at 3:39:10 PM UTC-4, mailsij...@gmail.com wrote:
>
> Hi,
>
> The machine that I run logstash process has two network interfaces (public 
> and private) and it looks like the logstash process picks up my private IP 
> address as the "publish_address" by default. As I use the option "node", it 
> is not able to join the cluster as this address is not visible externally. 
> Master node is in a different network. Could someone suggest how can I 
> specifically set the publish address for the logstash process? I know this 
> option is available for the elastic search processes in its config file 
> (elasticsearch.yml) but not aware of how to set this for the logstash 
> process.
>
> Thanks
> Sijo Mathew
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/ec56feca-62db-49e5-8e50-a29880cc3da9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Term suggester example?

2015-04-02 Thread Craig Ching

I am trying to get term suggest to work and it may be that it's not going 
to work the way I think it should, but I can't seem to get it to work *at 
all*.  So I put together an example index to illustrate the point, 
hopefully someone can point out what I'm doing wrong.

Basically, I want to be able to get suggestions for documents based on 
terms.  For instance, if I have documents with a name of "North America" 
and "South America", I would like a _suggest of "america" to return 
suggestions for both of those documents.  Note that I'm moving away from 
completion suggest because they are documented and appear to work only as 
prefix suggestors.

Here is my not-working example, appreciate any pointers!

#!/bin/sh

ES=localhost:9200

echo ">>> Deleting _all"
curl -XDELETE $ES/_all

CONTINENTS=("North America" "South America" "Europe" "Asia" "Australia")
for ((i = 0; i < ${#CONTINENTS[@]}; i++))
do
c=${CONTINENTS[$i]}
echo ">>> Indexing the '$c' document"
curl -XPOST $ES/continents/continent -d'
{
"name": "$c"
}
' && echo
echo "${CONTINENTS[$i]}"
done

curl -XPOST $ES/continents/_flush && echo

# Search for the document using the name
echo
echo ">>> Getting _suggest for 'america'"
echo
curl $ES/continents/_suggest -d'
{
"my-suggestion" : {
"text" : "america",
"term" : {
"field" : "name",
"suggest_mode" : "always"
}
}
}' && echo


Here's the output I'm currently getting:

>>> Getting _suggest for 'america'

{"_shards":{"total":5,"successful":5,"failed":0},"my-suggestion":[{"text":
"america","offset":0,"length":7,"options":[]}]}



-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/56349847-7ee6-4d06-9b26-d624878a311a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Problem with ES- Shield

2015-04-02 Thread Costin Leau

The configuration example is correct - one can pass any configuration property there and it will be picked up as 
explained here [1]


Additionally, to verify one can turn on logging [2] to see the connection is 
properly authenticated

[1] 
http://www.elastic.co/guide/en/elasticsearch/hadoop/master/pig.html#pig-configuration
[2] http://www.elastic.co/guide/en/elasticsearch/hadoop/master/logging.html

On 4/2/15 9:21 PM, Jay Modi wrote:

Still not sure what's going on with Marvel, unfortunately. No idea if this will 
help, but have you tried clearing your
browser cache or using a incognito window?

I think something like this:

TBLPROPERTIES('es.resource' = 't/t',
'es.nodes'='192.168.238.130:9200 ',
 'es.net.http.auth.user'='omar',
 'es.net.http.auth.pass'='omar')

I copy/pasted the wrong names earlier. I'm not a hadoop user but I believe that 
should work based off of the following
documentation:
http://www.elastic.co/guide/en/shield/current/hadoop.html
http://www.elastic.co/guide/en/elasticsearch/hadoop/current/hive.html#hive-configuration

On Thursday, April 2, 2015 at 2:05:15 PM UTC-4, BEN SALEM Omar wrote:

Jay,
when it comes to adding the .user and .password in the tblproperties
Do you know what exactly to do?
let's suppose that my admin esuser is omar and pwd omar
what should I do?
TBLPROPERTIES('es.resource' = 't/t',
'es.nodes'='192.168.238.130:9200 ',
 and now what to write);
and is there any further config to do?
Thanks for your time
ᐧ

On Thu, Apr 2, 2015 at 7:29 PM, BEN SALEM Omar > wrote:

Yes I had,
I used to manage my cluster with marvel perfectly and all the indices 
used to show up.
But now, none of them seem to be found due to time span; and even when I removed 
the shield this "porblem" occured.
I don't see how to "restore" the previous configuration.
Any idea?
ᐧ

On Thu, Apr 2, 2015 at 7:14 PM, Jay Modi > wrote:

There should not be anything extra necessary to work with Marvel. 
The browser prompted you for credentials
and you entered admin credentials, correct?

On Thursday, April 2, 2015 at 12:39:03 PM UTC-4, BEN SALEM Omar 
wrote:

Or not, that wenti as I'd like to go :))
Here is the last remaining issue :



ᐧ

On Thu, Apr 2, 2015 at 6:32 PM, BEN SALEM Omar 
 wrote:

That would be a problem !
Last question (as for now :p ), how to change the marvel 
timepicker? I can't see any indice of
mine(as I already told you)
And hey, thank you for your cooperation Jay
ᐧ

On Thu, Apr 2, 2015 at 6:26 PM, Jay Modi  
wrote:

Hi Omar,

I think you need to add "es.net.http.auth.user" and 
"es.net.http.auth_password" as
additional TBLPROPERTIES. You will need to using 
elasticsearch Hadoop 2.1 beta 3 for this to
work as far as I know.

-Jay

On Thursday, April 2, 2015 at 11:18:12 AM UTC-4, BEN 
SALEM Omar wrote:

Hi Jay,
Now that I have shield in both ES and Kibana, and I 
want to load some data from my hadoop
cluster to ES.
I usually do this :


create external table es_word_count(word string, 
count int)
STORED BY 
'org.elasticsearch.hadoop.__hive__.EsStorageHandler'
TBLPROPERTIES('es.resource' = 'test/test',
'es.nodes'='192.168.238.130 
:__92__00');
INSERT OVERWRITE TABLE es_word_count SELECT s.word, 
s.count  FROM word_counts1 s;


what should I do or add to access to my ES cluster 
as the esuser I've created?
Thanks
ᐧ

On Thu, Apr 2, 2015 at 1:08 PM, Jay Modi 
 wrote:

Hi Omar,

Have you configured Kibana and Marvel to work 
with Shield? For Kibana 4, please see

http://www.elastic.co/__guid__e/en/kibana/current/__production__.html#configuring-__kibana-__shield


 and

http://www.elastic.co/__guid__e/en/shield/current/___shield___with_kibana_4.html

Setting publish_address for logstash

2015-04-02 Thread mailsijomathew

Hi,

The machine that I run logstash process has two network interfaces (public 
and private) and it looks like the logstash process picks up my private IP 
address as the "publish_address" by default. As I use the option "node", it 
is not able to join the cluster as this address is not visible externally. 
Master node is in a different network. Could someone suggest how can I 
specifically set the publish address for the logstash process? I know this 
option is available for the elastic search processes in its config file 
(elasticsearch.yml) but not aware of how to set this for the logstash 
process.

Thanks
Sijo Mathew

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b9fda42d-8036-4fbc-9fcb-e91dfce5ce1d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: alias not indexing all documents

2015-04-02 Thread Stefanie

Hi,

I have found what the issue is.  In the version of elasticsearch that I am 
running the "now()" in the alias is evaluated at alias creation not at 
alias run time per:
https://github.com/elastic/elasticsearch/issues/8474.

As we are not going to upgrade our production system at this time does 
anyone know a way to work around this?

Thanks,
Stefanie

On Wednesday, April 1, 2015 at 5:28:23 PM UTC-4, Stefanie wrote:
>
> Hi,
>
> I have created a gist with an example of what my alias and search filter 
> look like.
> https://gist.github.com/swarzech/4e2bcdcc046587fb2845
>
> Thanks,
> Stefanie
>
> On Wednesday, April 1, 2015 at 5:15:25 PM UTC-4, Mark Walkom wrote:
>>
>> It'd help if you provided a gist/pastebin/etc link to the alias and the 
>> filter.
>>
>> On 2 April 2015 at 02:51, Stefanie  wrote:
>>
>>> Hi,
>>>
>>> I have an alias with filters that does not seem to be indexing any new 
>>> documents.  When I run the filter directly on the index I get the correct 
>>> results but when I call the alias the new documents are not being pulled.  
>>>
>>> This only seems to be happening with one alias as all of my other 
>>> aliases that work the same way don't seem to be having a problem.  I have 
>>> tried deleting the alias and recreating it but this didn't fix the issue.
>>>
>>> Does anyone know why this might be happening?
>>>
>>> Thank you,
>>> Stefanie
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/elasticsearch/48a5267d-7fa4-4d6e-bacb-54b93775955a%40googlegroups.com
>>>  
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/24670003-224c-4c7f-8da1-0a72211890b8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Alert notification with percolator

2015-04-02 Thread Lincoln Xiong

I try to use elasticsearch as a 2nd log output storage, to analyze some
info in logs. In this case, an alert triggers will be very useful. I read
through docs talking about percolator and I think this should be the way to
make it out.But after some trying, I found that I don't really get how
percolator works. It seems that I use REST API to index a document with a
percolator already set up, it will return if that document match the
percolator query or not. For my case, I use Logstash as input which of
course don't have this kind of feedback. And a "count" appears to be
accessible from REST that I can use to get this kind "feedback" from
percolator but I find it no where.

Could some one give me an idea about how I can achieve this kind of feature
with Elasticsearch?

I know I can have ways to trigger an alert in Logstash but for my case
Logstash is a temporary tool to input the data, I could possible not use it
in the future.
I also notice that Graylog has kind of alert. When the input event match
some keywords, the alarm will trigger. I guess it also use some percolator
APIs but I wish to know how can I do this alone with Elasticsearch only.

Thanks a lot.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/cf5da9d8-7000-4a0b-94af-3ce064feee90%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Problem with ES- Shield

Still not sure what's going on with Marvel, unfortunately. No idea if this 
will help, but have you tried clearing your browser cache or using a 
incognito window?

I think something like this:

TBLPROPERTIES('es.resource' = 't/t',
'es.nodes'='192.168.238.130:9200',
'es.net.http.auth.user'='omar',
'es.net.http.auth.pass'='omar')

I copy/pasted the wrong names earlier. I'm not a hadoop user but I believe 
that should work based off of the following documentation:
http://www.elastic.co/guide/en/shield/current/hadoop.html
http://www.elastic.co/guide/en/elasticsearch/hadoop/current/hive.html#hive-configuration

On Thursday, April 2, 2015 at 2:05:15 PM UTC-4, BEN SALEM Omar wrote:
>
> Jay,
> when it comes to adding the .user and .password in the tblproperties
> Do you know what exactly to do?
> let's suppose that my admin esuser is omar and pwd omar
> what should I do? 
> TBLPROPERTIES('es.resource' = 't/t',
> 'es.nodes'='192.168.238.130:9200',
> and now what to write);
> and is there any further config to do?
> Thanks for your time
> ᐧ
>
> On Thu, Apr 2, 2015 at 7:29 PM, BEN SALEM Omar  > wrote:
>
>> Yes I had,
>> I used to manage my cluster with marvel perfectly and all the indices 
>> used to show up.
>> But now, none of them seem to be found due to time span; and even when I 
>> removed the shield this "porblem" occured.
>> I don't see how to "restore" the previous configuration.
>> Any idea?
>> ᐧ
>>
>> On Thu, Apr 2, 2015 at 7:14 PM, Jay Modi > 
>> wrote:
>>
>>> There should not be anything extra necessary to work with Marvel. The 
>>> browser prompted you for credentials and you entered admin credentials, 
>>> correct?
>>>
>>> On Thursday, April 2, 2015 at 12:39:03 PM UTC-4, BEN SALEM Omar wrote:

 Or not, that wenti as I'd like to go :))
 Here is the last remaining issue :


 
 ᐧ

 On Thu, Apr 2, 2015 at 6:32 PM, BEN SALEM Omar  
 wrote:

> That would be a problem ! 
> Last question (as for now :p ), how to change the marvel timepicker? I 
> can't see any indice of mine(as I already told you)
> And hey, thank you for your cooperation Jay
> ᐧ
>
> On Thu, Apr 2, 2015 at 6:26 PM, Jay Modi  wrote:
>
>> Hi Omar,
>>
>> I think you need to add "es.net.http.auth.user" and 
>> "es.net.http.auth_password" as additional TBLPROPERTIES. You will need 
>> to 
>> using elasticsearch Hadoop 2.1 beta 3 for this to work as far as I know.
>>
>> -Jay
>>
>> On Thursday, April 2, 2015 at 11:18:12 AM UTC-4, BEN SALEM Omar wrote:
>>>
>>> Hi Jay,
>>> Now that I have shield in both ES and Kibana, and I want to load 
>>> some data from my hadoop cluster to ES.
>>> I usually do this : 
>>> 
>>> 
>>> create external table es_word_count(word string, count int)
>>> STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler' 
>>> TBLPROPERTIES('es.resource' = 'test/test',
>>> 'es.nodes'='192.168.238.130:9200');
>>>  
>>> INSERT OVERWRITE TABLE es_word_count SELECT s.word, s.count  FROM 
>>> word_counts1 s;
>>>
>>>
>>> what should I do or add to access to my ES cluster as the esuser 
>>> I've created? 
>>> Thanks
>>> ᐧ
>>>
>>> On Thu, Apr 2, 2015 at 1:08 PM, Jay Modi  wrote:
>>>
 Hi Omar,

 Have you configured Kibana and Marvel to work with Shield? For 
 Kibana 4, please see http://www.elastic.co/guide/en/kibana/current/
 production.html#configuring-kibana-shield and 
 http://www.elastic.co/guide/en/shield/current/_shield_
 with_kibana_4.html. For Marvel, see http://www.elastic.co/guid
 e/en/shield/current/marvel.html

 -Jay

 On Thursday, April 2, 2015 at 6:45:55 AM UTC-4, BEN SALEM Omar 
 wrote:
>
> Hi Jay,
>
> I have a question, now that I've added an esuser as an admin, I 
> can not run my Kibana anymore, 
>
>
> Any idea on what to do to overcome this?
>
> Another basic question, when I open the marvel, it shows no indice 
> because the sample of data I'm working with is in "2014", how to 
> change my 
> time span to see my indices again?
> btwn, before I install the shield, Marvel used to recognize my 
> timestamp by default
>
> 
> ᐧ
> Thank you,
> Omar,
>
> On Wed, Apr 1, 2015 at 8:22 PM, BEN SALEM Omar <
> omar.b...@esprit.tn> wrote:
>
>> When I try to add an esuser and grant him a role:
>>
>> bin/shield/esusers useradd rdeniro -p taxidriver -r admin
>>
>> It says that there is an error in the "esusers" file in the 
>> elef['$ES_INCLUDE"] line, I think the line 31 or 38, I forgot.
>> In the line above, there is a $...done: w

Re: Problem with ES- Shield

2015-04-02 Thread 'BEN SALEM Omar' via elasticsearch

Jay,
when it comes to adding the .user and .password in the tblproperties
Do you know what exactly to do?
let's suppose that my admin esuser is omar and pwd omar
what should I do?
TBLPROPERTIES('es.resource' = 't/t',
'es.nodes'='192.168.238.130:9200',
and now what to write);
and is there any further config to do?
Thanks for your time
ᐧ

On Thu, Apr 2, 2015 at 7:29 PM, BEN SALEM Omar 
wrote:

> Yes I had,
> I used to manage my cluster with marvel perfectly and all the indices used
> to show up.
> But now, none of them seem to be found due to time span; and even when I
> removed the shield this "porblem" occured.
> I don't see how to "restore" the previous configuration.
> Any idea?
> ᐧ
>
> On Thu, Apr 2, 2015 at 7:14 PM, Jay Modi  wrote:
>
>> There should not be anything extra necessary to work with Marvel. The
>> browser prompted you for credentials and you entered admin credentials,
>> correct?
>>
>> On Thursday, April 2, 2015 at 12:39:03 PM UTC-4, BEN SALEM Omar wrote:
>>>
>>> Or not, that wenti as I'd like to go :))
>>> Here is the last remaining issue :
>>>
>>>
>>> 
>>> ᐧ
>>>
>>> On Thu, Apr 2, 2015 at 6:32 PM, BEN SALEM Omar 
>>> wrote:
>>>
 That would be a problem !
 Last question (as for now :p ), how to change the marvel timepicker? I
 can't see any indice of mine(as I already told you)
 And hey, thank you for your cooperation Jay
 ᐧ

 On Thu, Apr 2, 2015 at 6:26 PM, Jay Modi  wrote:

> Hi Omar,
>
> I think you need to add "es.net.http.auth.user" and
> "es.net.http.auth_password" as additional TBLPROPERTIES. You will need to
> using elasticsearch Hadoop 2.1 beta 3 for this to work as far as I know.
>
> -Jay
>
> On Thursday, April 2, 2015 at 11:18:12 AM UTC-4, BEN SALEM Omar wrote:
>>
>> Hi Jay,
>> Now that I have shield in both ES and Kibana, and I want to load some
>> data from my hadoop cluster to ES.
>> I usually do this :
>> 
>> 
>> create external table es_word_count(word string, count int)
>> STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
>> TBLPROPERTIES('es.resource' = 'test/test',
>> 'es.nodes'='192.168.238.130:9200');
>>
>> INSERT OVERWRITE TABLE es_word_count SELECT s.word, s.count  FROM
>> word_counts1 s;
>>
>>
>> what should I do or add to access to my ES cluster as the esuser I've
>> created?
>> Thanks
>> ᐧ
>>
>> On Thu, Apr 2, 2015 at 1:08 PM, Jay Modi  wrote:
>>
>>> Hi Omar,
>>>
>>> Have you configured Kibana and Marvel to work with Shield? For
>>> Kibana 4, please see http://www.elastic.co/guide/en/kibana/current/
>>> production.html#configuring-kibana-shield and http://www.elastic.co/
>>> guide/en/shield/current/_shield_with_kibana_4.html. For Marvel, see
>>> http://www.elastic.co/guide/en/shield/current/marvel.html
>>>
>>> -Jay
>>>
>>> On Thursday, April 2, 2015 at 6:45:55 AM UTC-4, BEN SALEM Omar wrote:

 Hi Jay,

 I have a question, now that I've added an esuser as an admin, I can
 not run my Kibana anymore,


 Any idea on what to do to overcome this?

 Another basic question, when I open the marvel, it shows no indice
 because the sample of data I'm working with is in "2014", how to 
 change my
 time span to see my indices again?
 btwn, before I install the shield, Marvel used to recognize my
 timestamp by default

 
 ᐧ
 Thank you,
 Omar,

 On Wed, Apr 1, 2015 at 8:22 PM, BEN SALEM Omar >>> > wrote:

> When I try to add an esuser and grant him a role:
>
> bin/shield/esusers useradd rdeniro -p taxidriver -r admin
>
> It says that there is an error in the "esusers" file in the
> elef['$ES_INCLUDE"] line, I think the line 31 or 38, I forgot.
> In the line above, there is a $...done: which ends with ":", if
> you change it to done; with ";" , the error doesn't show up anymore.
>
> I'm installing the whole thing in the Hortonworks Sandbox 2.0 vm
>
> On Wed, Apr 1, 2015 at 8:14 PM, Jay Modi  wrote:
>
>> I'm glad you were able to fix this issue. I have a few questions
>> that may help us ensure this fix is pulled back into Shield. What OS 
>> and
>> Shell are you using? Which line in the script did you change 
>> exactly? I'm
>> looking at the script right now and I don't see any instance of 
>> "done:"
>>
>> Just an FYI, there is another way to handle the configuration
>> directory issue that is documented in the configuring your 
>> environment
>> section of the shield guide:
>> http://www.elastic.co/guide/en/shield/current/getting-starte
>> d.html

Support multiple data sources

2015-04-02 Thread Yosi Attias

Hi,

I am currently using kibana and I really like it for dashboard - but most 
of my data is in the mssql or more than 3 elastic indexes.
I would like to create one dashboard that gets data from mssql and from my 
other elastic indexes.


Is this possible today? If not, kibana core committers I will be more than 
happy to create a PR for this, will you apply it?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e00336d1-26d5-4113-8eb8-b50d77ab26ee%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Problem with ES- Shield

2015-04-02 Thread 'BEN SALEM Omar' via elasticsearch

Yes I had,
I used to manage my cluster with marvel perfectly and all the indices used
to show up.
But now, none of them seem to be found due to time span; and even when I
removed the shield this "porblem" occured.
I don't see how to "restore" the previous configuration.
Any idea?
ᐧ

On Thu, Apr 2, 2015 at 7:14 PM, Jay Modi  wrote:

> There should not be anything extra necessary to work with Marvel. The
> browser prompted you for credentials and you entered admin credentials,
> correct?
>
> On Thursday, April 2, 2015 at 12:39:03 PM UTC-4, BEN SALEM Omar wrote:
>>
>> Or not, that wenti as I'd like to go :))
>> Here is the last remaining issue :
>>
>>
>> 
>> ᐧ
>>
>> On Thu, Apr 2, 2015 at 6:32 PM, BEN SALEM Omar 
>> wrote:
>>
>>> That would be a problem !
>>> Last question (as for now :p ), how to change the marvel timepicker? I
>>> can't see any indice of mine(as I already told you)
>>> And hey, thank you for your cooperation Jay
>>> ᐧ
>>>
>>> On Thu, Apr 2, 2015 at 6:26 PM, Jay Modi  wrote:
>>>
 Hi Omar,

 I think you need to add "es.net.http.auth.user" and
 "es.net.http.auth_password" as additional TBLPROPERTIES. You will need to
 using elasticsearch Hadoop 2.1 beta 3 for this to work as far as I know.

 -Jay

 On Thursday, April 2, 2015 at 11:18:12 AM UTC-4, BEN SALEM Omar wrote:
>
> Hi Jay,
> Now that I have shield in both ES and Kibana, and I want to load some
> data from my hadoop cluster to ES.
> I usually do this :
> 
> 
> create external table es_word_count(word string, count int)
> STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
> TBLPROPERTIES('es.resource' = 'test/test',
> 'es.nodes'='192.168.238.130:9200');
>
> INSERT OVERWRITE TABLE es_word_count SELECT s.word, s.count  FROM
> word_counts1 s;
>
>
> what should I do or add to access to my ES cluster as the esuser I've
> created?
> Thanks
> ᐧ
>
> On Thu, Apr 2, 2015 at 1:08 PM, Jay Modi  wrote:
>
>> Hi Omar,
>>
>> Have you configured Kibana and Marvel to work with Shield? For Kibana
>> 4, please see http://www.elastic.co/guide/en/kibana/current/
>> production.html#configuring-kibana-shield and http://www.elastic.co/
>> guide/en/shield/current/_shield_with_kibana_4.html. For Marvel, see
>> http://www.elastic.co/guide/en/shield/current/marvel.html
>>
>> -Jay
>>
>> On Thursday, April 2, 2015 at 6:45:55 AM UTC-4, BEN SALEM Omar wrote:
>>>
>>> Hi Jay,
>>>
>>> I have a question, now that I've added an esuser as an admin, I can
>>> not run my Kibana anymore,
>>>
>>>
>>> Any idea on what to do to overcome this?
>>>
>>> Another basic question, when I open the marvel, it shows no indice
>>> because the sample of data I'm working with is in "2014", how to change 
>>> my
>>> time span to see my indices again?
>>> btwn, before I install the shield, Marvel used to recognize my
>>> timestamp by default
>>>
>>> 
>>> ᐧ
>>> Thank you,
>>> Omar,
>>>
>>> On Wed, Apr 1, 2015 at 8:22 PM, BEN SALEM Omar 
>>> wrote:
>>>
 When I try to add an esuser and grant him a role:

 bin/shield/esusers useradd rdeniro -p taxidriver -r admin

 It says that there is an error in the "esusers" file in the
 elef['$ES_INCLUDE"] line, I think the line 31 or 38, I forgot.
 In the line above, there is a $...done: which ends with ":", if you
 change it to done; with ";" , the error doesn't show up anymore.

 I'm installing the whole thing in the Hortonworks Sandbox 2.0 vm

 On Wed, Apr 1, 2015 at 8:14 PM, Jay Modi  wrote:

> I'm glad you were able to fix this issue. I have a few questions
> that may help us ensure this fix is pulled back into Shield. What OS 
> and
> Shell are you using? Which line in the script did you change exactly? 
> I'm
> looking at the script right now and I don't see any instance of 
> "done:"
>
> Just an FYI, there is another way to handle the configuration
> directory issue that is documented in the configuring your environment
> section of the shield guide:
> http://www.elastic.co/guide/en/shield/current/getting-starte
> d.html#_configuring_your_environment
>
> On Wednesday, April 1, 2015 at 2:07:28 PM UTC-4, BEN SALEM Omar
> wrote:
>>
>> Ok, I've managed to solve my own problem.
>>
>> First of all things, if you want to just remove the shield plugin
>> run under the elasticsearch :
>>
>> $ bin/plugin --remove shield
>>
>> Now, to the syntax error in the elef line in the "esusers",
>> generally, when there is such an error, it's related to things 
>> before that

Re: Problem with ES- Shield

There should not be anything extra necessary to work with Marvel. The 
browser prompted you for credentials and you entered admin credentials, 
correct?

On Thursday, April 2, 2015 at 12:39:03 PM UTC-4, BEN SALEM Omar wrote:
>
> Or not, that wenti as I'd like to go :))
> Here is the last remaining issue :
>
>
> 
> ᐧ
>
> On Thu, Apr 2, 2015 at 6:32 PM, BEN SALEM Omar  > wrote:
>
>> That would be a problem ! 
>> Last question (as for now :p ), how to change the marvel timepicker? I 
>> can't see any indice of mine(as I already told you)
>> And hey, thank you for your cooperation Jay
>> ᐧ
>>
>> On Thu, Apr 2, 2015 at 6:26 PM, Jay Modi > 
>> wrote:
>>
>>> Hi Omar,
>>>
>>> I think you need to add "es.net.http.auth.user" and 
>>> "es.net.http.auth_password" as additional TBLPROPERTIES. You will need to 
>>> using elasticsearch Hadoop 2.1 beta 3 for this to work as far as I know.
>>>
>>> -Jay
>>>
>>> On Thursday, April 2, 2015 at 11:18:12 AM UTC-4, BEN SALEM Omar wrote:

 Hi Jay,
 Now that I have shield in both ES and Kibana, and I want to load some 
 data from my hadoop cluster to ES.
 I usually do this : 
 
 
 create external table es_word_count(word string, count int)
 STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler' 
 TBLPROPERTIES('es.resource' = 'test/test',
 'es.nodes'='192.168.238.130:9200');
  
 INSERT OVERWRITE TABLE es_word_count SELECT s.word, s.count  FROM 
 word_counts1 s;


 what should I do or add to access to my ES cluster as the esuser I've 
 created? 
 Thanks
 ᐧ

 On Thu, Apr 2, 2015 at 1:08 PM, Jay Modi  wrote:

> Hi Omar,
>
> Have you configured Kibana and Marvel to work with Shield? For Kibana 
> 4, please see http://www.elastic.co/guide/en/kibana/current/
> production.html#configuring-kibana-shield and http://www.elastic.co/
> guide/en/shield/current/_shield_with_kibana_4.html. For Marvel, see 
> http://www.elastic.co/guide/en/shield/current/marvel.html
>
> -Jay
>
> On Thursday, April 2, 2015 at 6:45:55 AM UTC-4, BEN SALEM Omar wrote:
>>
>> Hi Jay,
>>
>> I have a question, now that I've added an esuser as an admin, I can 
>> not run my Kibana anymore, 
>>
>>
>> Any idea on what to do to overcome this?
>>
>> Another basic question, when I open the marvel, it shows no indice 
>> because the sample of data I'm working with is in "2014", how to change 
>> my 
>> time span to see my indices again?
>> btwn, before I install the shield, Marvel used to recognize my 
>> timestamp by default
>>
>> 
>> ᐧ
>> Thank you,
>> Omar,
>>
>> On Wed, Apr 1, 2015 at 8:22 PM, BEN SALEM Omar  
>> wrote:
>>
>>> When I try to add an esuser and grant him a role:
>>>
>>> bin/shield/esusers useradd rdeniro -p taxidriver -r admin
>>>
>>> It says that there is an error in the "esusers" file in the 
>>> elef['$ES_INCLUDE"] line, I think the line 31 or 38, I forgot.
>>> In the line above, there is a $...done: which ends with ":", if you 
>>> change it to done; with ";" , the error doesn't show up anymore.
>>>
>>> I'm installing the whole thing in the Hortonworks Sandbox 2.0 vm
>>>
>>> On Wed, Apr 1, 2015 at 8:14 PM, Jay Modi  wrote:
>>>
 I'm glad you were able to fix this issue. I have a few questions 
 that may help us ensure this fix is pulled back into Shield. What OS 
 and 
 Shell are you using? Which line in the script did you change exactly? 
 I'm 
 looking at the script right now and I don't see any instance of "done:"

 Just an FYI, there is another way to handle the configuration 
 directory issue that is documented in the configuring your environment 
 section of the shield guide:
 http://www.elastic.co/guide/en/shield/current/getting-starte
 d.html#_configuring_your_environment

 On Wednesday, April 1, 2015 at 2:07:28 PM UTC-4, BEN SALEM Omar 
 wrote:
>
> Ok, I've managed to solve my own problem.
>
> First of all things, if you want to just remove the shield plugin 
> run under the elasticsearch :
>
> $ bin/plugin --remove shield
>
> Now, to the syntax error in the elef line in the "esusers", 
> generally, when there is such an error, it's related to things before 
> that 
> line.
> In fact there was something like this : 
> $done: 
> correction : 
> $done;
>
> Now, we could add users within shield and grunt them the desired 
> role.
>
> But, even if I've done it, I couldn't access to my ES cluster with 
> the "new born user"; I think because my ES is running as a service 
> and not 
> starting with

Re: Problem with ES- Shield

2015-04-02 Thread 'BEN SALEM Omar' via elasticsearch

Or not, that wenti as I'd like to go :))
Here is the last remaining issue :



ᐧ

On Thu, Apr 2, 2015 at 6:32 PM, BEN SALEM Omar 
wrote:

> That would be a problem !
> Last question (as for now :p ), how to change the marvel timepicker? I
> can't see any indice of mine(as I already told you)
> And hey, thank you for your cooperation Jay
> ᐧ
>
> On Thu, Apr 2, 2015 at 6:26 PM, Jay Modi  wrote:
>
>> Hi Omar,
>>
>> I think you need to add "es.net.http.auth.user" and
>> "es.net.http.auth_password" as additional TBLPROPERTIES. You will need to
>> using elasticsearch Hadoop 2.1 beta 3 for this to work as far as I know.
>>
>> -Jay
>>
>> On Thursday, April 2, 2015 at 11:18:12 AM UTC-4, BEN SALEM Omar wrote:
>>>
>>> Hi Jay,
>>> Now that I have shield in both ES and Kibana, and I want to load some
>>> data from my hadoop cluster to ES.
>>> I usually do this :
>>> 
>>> 
>>> create external table es_word_count(word string, count int)
>>> STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
>>> TBLPROPERTIES('es.resource' = 'test/test',
>>> 'es.nodes'='192.168.238.130:9200');
>>>
>>> INSERT OVERWRITE TABLE es_word_count SELECT s.word, s.count  FROM
>>> word_counts1 s;
>>>
>>>
>>> what should I do or add to access to my ES cluster as the esuser I've
>>> created?
>>> Thanks
>>> ᐧ
>>>
>>> On Thu, Apr 2, 2015 at 1:08 PM, Jay Modi  wrote:
>>>
 Hi Omar,

 Have you configured Kibana and Marvel to work with Shield? For Kibana
 4, please see http://www.elastic.co/guide/en/kibana/current/
 production.html#configuring-kibana-shield and http://www.elastic.co/
 guide/en/shield/current/_shield_with_kibana_4.html. For Marvel, see
 http://www.elastic.co/guide/en/shield/current/marvel.html

 -Jay

 On Thursday, April 2, 2015 at 6:45:55 AM UTC-4, BEN SALEM Omar wrote:
>
> Hi Jay,
>
> I have a question, now that I've added an esuser as an admin, I can
> not run my Kibana anymore,
>
>
> Any idea on what to do to overcome this?
>
> Another basic question, when I open the marvel, it shows no indice
> because the sample of data I'm working with is in "2014", how to change my
> time span to see my indices again?
> btwn, before I install the shield, Marvel used to recognize my
> timestamp by default
>
> 
> ᐧ
> Thank you,
> Omar,
>
> On Wed, Apr 1, 2015 at 8:22 PM, BEN SALEM Omar 
> wrote:
>
>> When I try to add an esuser and grant him a role:
>>
>> bin/shield/esusers useradd rdeniro -p taxidriver -r admin
>>
>> It says that there is an error in the "esusers" file in the
>> elef['$ES_INCLUDE"] line, I think the line 31 or 38, I forgot.
>> In the line above, there is a $...done: which ends with ":", if you
>> change it to done; with ";" , the error doesn't show up anymore.
>>
>> I'm installing the whole thing in the Hortonworks Sandbox 2.0 vm
>>
>> On Wed, Apr 1, 2015 at 8:14 PM, Jay Modi  wrote:
>>
>>> I'm glad you were able to fix this issue. I have a few questions
>>> that may help us ensure this fix is pulled back into Shield. What OS and
>>> Shell are you using? Which line in the script did you change exactly? 
>>> I'm
>>> looking at the script right now and I don't see any instance of "done:"
>>>
>>> Just an FYI, there is another way to handle the configuration
>>> directory issue that is documented in the configuring your environment
>>> section of the shield guide:
>>> http://www.elastic.co/guide/en/shield/current/getting-starte
>>> d.html#_configuring_your_environment
>>>
>>> On Wednesday, April 1, 2015 at 2:07:28 PM UTC-4, BEN SALEM Omar
>>> wrote:

 Ok, I've managed to solve my own problem.

 First of all things, if you want to just remove the shield plugin
 run under the elasticsearch :

 $ bin/plugin --remove shield

 Now, to the syntax error in the elef line in the "esusers",
 generally, when there is such an error, it's related to things before 
 that
 line.
 In fact there was something like this :
 $done:
 correction :
 $done;

 Now, we could add users within shield and grunt them the desired
 role.

 But, even if I've done it, I couldn't access to my ES cluster with
 the "new born user"; I think because my ES is running as a service and 
 not
 starting with bin/elasticsearch, that lead to "misunderstanding."

 The solution I found was to transfer the /usr/share/elasticsearch/
 config/shield under /etc/elasticsearch
 $ln -s /usr/share/elasticsearch/config/shield/ /etc/elasticsearch

 And that was it !
 Hope that could be of a help for anyone else facing the same
 problem !

 On Wednesday, April 1, 20

Re: Problem with ES- Shield

2015-04-02 Thread 'BEN SALEM Omar' via elasticsearch

That would be a problem !
Last question (as for now :p ), how to change the marvel timepicker? I
can't see any indice of mine(as I already told you)
And hey, thank you for your cooperation Jay
ᐧ

On Thu, Apr 2, 2015 at 6:26 PM, Jay Modi  wrote:

> Hi Omar,
>
> I think you need to add "es.net.http.auth.user" and
> "es.net.http.auth_password" as additional TBLPROPERTIES. You will need to
> using elasticsearch Hadoop 2.1 beta 3 for this to work as far as I know.
>
> -Jay
>
> On Thursday, April 2, 2015 at 11:18:12 AM UTC-4, BEN SALEM Omar wrote:
>>
>> Hi Jay,
>> Now that I have shield in both ES and Kibana, and I want to load some
>> data from my hadoop cluster to ES.
>> I usually do this :
>> 
>> 
>> create external table es_word_count(word string, count int)
>> STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
>> TBLPROPERTIES('es.resource' = 'test/test',
>> 'es.nodes'='192.168.238.130:9200');
>>
>> INSERT OVERWRITE TABLE es_word_count SELECT s.word, s.count  FROM
>> word_counts1 s;
>>
>>
>> what should I do or add to access to my ES cluster as the esuser I've
>> created?
>> Thanks
>> ᐧ
>>
>> On Thu, Apr 2, 2015 at 1:08 PM, Jay Modi  wrote:
>>
>>> Hi Omar,
>>>
>>> Have you configured Kibana and Marvel to work with Shield? For Kibana 4,
>>> please see http://www.elastic.co/guide/en/kibana/current/
>>> production.html#configuring-kibana-shield and http://www.elastic.co/
>>> guide/en/shield/current/_shield_with_kibana_4.html. For Marvel, see
>>> http://www.elastic.co/guide/en/shield/current/marvel.html
>>>
>>> -Jay
>>>
>>> On Thursday, April 2, 2015 at 6:45:55 AM UTC-4, BEN SALEM Omar wrote:

 Hi Jay,

 I have a question, now that I've added an esuser as an admin, I can not
 run my Kibana anymore,


 Any idea on what to do to overcome this?

 Another basic question, when I open the marvel, it shows no indice
 because the sample of data I'm working with is in "2014", how to change my
 time span to see my indices again?
 btwn, before I install the shield, Marvel used to recognize my
 timestamp by default

 
 ᐧ
 Thank you,
 Omar,

 On Wed, Apr 1, 2015 at 8:22 PM, BEN SALEM Omar 
 wrote:

> When I try to add an esuser and grant him a role:
>
> bin/shield/esusers useradd rdeniro -p taxidriver -r admin
>
> It says that there is an error in the "esusers" file in the
> elef['$ES_INCLUDE"] line, I think the line 31 or 38, I forgot.
> In the line above, there is a $...done: which ends with ":", if you
> change it to done; with ";" , the error doesn't show up anymore.
>
> I'm installing the whole thing in the Hortonworks Sandbox 2.0 vm
>
> On Wed, Apr 1, 2015 at 8:14 PM, Jay Modi  wrote:
>
>> I'm glad you were able to fix this issue. I have a few questions that
>> may help us ensure this fix is pulled back into Shield. What OS and Shell
>> are you using? Which line in the script did you change exactly? I'm 
>> looking
>> at the script right now and I don't see any instance of "done:"
>>
>> Just an FYI, there is another way to handle the configuration
>> directory issue that is documented in the configuring your environment
>> section of the shield guide:
>> http://www.elastic.co/guide/en/shield/current/getting-starte
>> d.html#_configuring_your_environment
>>
>> On Wednesday, April 1, 2015 at 2:07:28 PM UTC-4, BEN SALEM Omar wrote:
>>>
>>> Ok, I've managed to solve my own problem.
>>>
>>> First of all things, if you want to just remove the shield plugin
>>> run under the elasticsearch :
>>>
>>> $ bin/plugin --remove shield
>>>
>>> Now, to the syntax error in the elef line in the "esusers",
>>> generally, when there is such an error, it's related to things before 
>>> that
>>> line.
>>> In fact there was something like this :
>>> $done:
>>> correction :
>>> $done;
>>>
>>> Now, we could add users within shield and grunt them the desired
>>> role.
>>>
>>> But, even if I've done it, I couldn't access to my ES cluster with
>>> the "new born user"; I think because my ES is running as a service and 
>>> not
>>> starting with bin/elasticsearch, that lead to "misunderstanding."
>>>
>>> The solution I found was to transfer the /usr/share/elasticsearch/
>>> config/shield under /etc/elasticsearch
>>> $ln -s /usr/share/elasticsearch/config/shield/ /etc/elasticsearch
>>>
>>> And that was it !
>>> Hope that could be of a help for anyone else facing the same problem
>>> !
>>>
>>> On Wednesday, April 1, 2015 at 5:28:14 PM UTC+2, BEN SALEM Omar
>>> wrote:

 I wanted to add authentication to my ES, thus I downloaded the
 shield :
 https://www.elastic.co/downloads/shield

 Then I've tried to add a user to give him the admin role as it's
>>

Re: Problem with ES- Shield

Hi Omar,

I think you need to add "es.net.http.auth.user" and 
"es.net.http.auth_password" as additional TBLPROPERTIES. You will need to 
using elasticsearch Hadoop 2.1 beta 3 for this to work as far as I know.

-Jay

On Thursday, April 2, 2015 at 11:18:12 AM UTC-4, BEN SALEM Omar wrote:
>
> Hi Jay,
> Now that I have shield in both ES and Kibana, and I want to load some data 
> from my hadoop cluster to ES.
> I usually do this : 
> 
> 
> create external table es_word_count(word string, count int)
> STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler' 
> TBLPROPERTIES('es.resource' = 'test/test',
> 'es.nodes'='192.168.238.130:9200');
>  
> INSERT OVERWRITE TABLE es_word_count SELECT s.word, s.count  FROM 
> word_counts1 s;
>
>
> what should I do or add to access to my ES cluster as the esuser I've 
> created? 
> Thanks
> ᐧ
>
> On Thu, Apr 2, 2015 at 1:08 PM, Jay Modi > 
> wrote:
>
>> Hi Omar,
>>
>> Have you configured Kibana and Marvel to work with Shield? For Kibana 4, 
>> please see 
>> http://www.elastic.co/guide/en/kibana/current/production.html#configuring-kibana-shield
>>  
>> and 
>> http://www.elastic.co/guide/en/shield/current/_shield_with_kibana_4.html. 
>> For Marvel, see http://www.elastic.co/guide/en/shield/current/marvel.html
>>
>> -Jay
>>
>> On Thursday, April 2, 2015 at 6:45:55 AM UTC-4, BEN SALEM Omar wrote:
>>>
>>> Hi Jay,
>>>
>>> I have a question, now that I've added an esuser as an admin, I can not 
>>> run my Kibana anymore, 
>>>
>>>
>>> Any idea on what to do to overcome this?
>>>
>>> Another basic question, when I open the marvel, it shows no indice 
>>> because the sample of data I'm working with is in "2014", how to change my 
>>> time span to see my indices again?
>>> btwn, before I install the shield, Marvel used to recognize my timestamp 
>>> by default
>>>
>>> 
>>> ᐧ
>>> Thank you,
>>> Omar,
>>>
>>> On Wed, Apr 1, 2015 at 8:22 PM, BEN SALEM Omar  
>>> wrote:
>>>
 When I try to add an esuser and grant him a role:

 bin/shield/esusers useradd rdeniro -p taxidriver -r admin

 It says that there is an error in the "esusers" file in the 
 elef['$ES_INCLUDE"] line, I think the line 31 or 38, I forgot.
 In the line above, there is a $...done: which ends with ":", if you 
 change it to done; with ";" , the error doesn't show up anymore.

 I'm installing the whole thing in the Hortonworks Sandbox 2.0 vm

 On Wed, Apr 1, 2015 at 8:14 PM, Jay Modi  wrote:

> I'm glad you were able to fix this issue. I have a few questions that 
> may help us ensure this fix is pulled back into Shield. What OS and Shell 
> are you using? Which line in the script did you change exactly? I'm 
> looking 
> at the script right now and I don't see any instance of "done:"
>
> Just an FYI, there is another way to handle the configuration 
> directory issue that is documented in the configuring your environment 
> section of the shield guide:
> http://www.elastic.co/guide/en/shield/current/getting-
> started.html#_configuring_your_environment
>
> On Wednesday, April 1, 2015 at 2:07:28 PM UTC-4, BEN SALEM Omar wrote:
>>
>> Ok, I've managed to solve my own problem.
>>
>> First of all things, if you want to just remove the shield plugin run 
>> under the elasticsearch :
>>
>> $ bin/plugin --remove shield
>>
>> Now, to the syntax error in the elef line in the "esusers", 
>> generally, when there is such an error, it's related to things before 
>> that 
>> line.
>> In fact there was something like this : 
>> $done: 
>> correction : 
>> $done;
>>
>> Now, we could add users within shield and grunt them the desired role.
>>
>> But, even if I've done it, I couldn't access to my ES cluster with 
>> the "new born user"; I think because my ES is running as a service and 
>> not 
>> starting with bin/elasticsearch, that lead to "misunderstanding."
>>
>> The solution I found was to transfer the /usr/share/elasticsearch/
>> config/shield under /etc/elasticsearch
>> $ln -s /usr/share/elasticsearch/config/shield/ /etc/elasticsearch
>>
>> And that was it ! 
>> Hope that could be of a help for anyone else facing the same problem !
>>
>> On Wednesday, April 1, 2015 at 5:28:14 PM UTC+2, BEN SALEM Omar wrote:
>>>
>>> I wanted to add authentication to my ES, thus I downloaded the 
>>> shield :
>>> https://www.elastic.co/downloads/shield
>>>
>>> Then I've tried to add a user to give him the admin role as it's 
>>> mentionned here :
>>> http://www.elastic.co/guide/en/shield/current/getting-started.html
>>> (I've added it with root, cause I start my es with root..)
>>>
>>> But when I try to do so, an error occure saying that there is a 
>>> syntax error in the 38 line "the elef line" 
>>> elef[$ES_INCLUDE] bla bla
>>> I've tried many

Re: Evaluating Moving to Discourse - Feedback Wanted

2015-04-02 Thread Jack Park

Leslie, this is a truly important conversation to have.

To anticipate my thoughts, I have been working on evolving open source
conversation platforms, using ElasticSearch. There are two aspects of my
work (others are doing similar work):

1- conversation is structured. In fact it is not simply an indented outline
conversation; each contribution (node) is akin to a tweet: a terse
statement or question about a single subject. Thus, trees grow along
subject lines. Each node is one of a question, a statement which can be an
answer to a question or an amplification on another answer, or a statement
which is a pro or con argument.

2- a topic map federates the topics in play, allows for relations among
those topics. The same apparatus allows for crosslinking nodes in
conversations simply because each conversation node is, in fact, its own
topic in the topic map, but each node is *about* a different specific
topic, say, "Yellow on startup". Thus, all that is knowable about "Yellow
on startup", every conversation node, the manual pages, blog pages,
whatever, is eventually captured by that specific node in the topic map
(document in ES). I recently gave a talk which speaks to my views on topic
maps in the context of using ElasticSearch in a much wider context [1].

To respond to your question, I think that Discourse is a pretty amazing
product. It might be that a topic map can be added to it to round out the
platform.  A topic map helps to increase the SNR (signal to noise ratio) in
a sense similar to the way people jump to Wikipedia to do a quick check on
what something is about. But more than that, a topic map behaves like a
"geolocator" for all the topics in play in conversations.

It would seem that email interaction is one of those required capabilities.
Doing structured conversations in email has not yet been shown to be easy.
My observation has been that users of the ElasticSearch email list have
been very disciplined, staying on topic; that's nowhere near the case in
many other lists. But, does that translate to implementation as a
structured branching-tree conversation?  Hard to say.

But, is email really required?  I notice that some people come to the email
list first with a question about this or that exception; I've found it to
be extremely useful to paste the exception in the search bar on my browser
and get a great answer before coming to the list, though there have been
(and still are) some sticky exceptions for which answers just are not
available online.

Nevertheless, this is an important conversation to have, and not just for
the ElasticSearch community; it seems to me there is a huge demand to
improve the SNR for many online communities; I see ElasticSearch in that
solution space.

Cheers
Jack
[1] http://www.slideshare.net/jackpark/jst-talk-final

On Thu, Apr 2, 2015 at 8:36 AM, leslie.hawthorn 
wrote:

> Hello everyone,
>
> As we’ve begun to scale up development on three different open source
> projects, we’ve found Google Groups to be a difficult solution for dealing
> with all of our needs for community support. We’ve got multiple mailing
> lists going, which can be confusing for new folks trying to figure out
> where to go to ask a question.
>
> We’ve also found our lists are becoming noisy in the “good problem to
> have” kind of way. As we’ve seen more user adoption, and across such a wide
> variety of use cases, we’re getting widely different types of questions
> asked. For example, I can imagine that folks not using our Python client
> would rather not be distracted with emails about it.
>
> There’s also a few other strikes against Groups as a tool, such as the
> fact that it is no longer a supported product by Google, it provides no API
> hooks and it is not available for users in China.
>
> We’ve evaluated several options and we’re currently considering shuttering
> the elasticsearch-user and logstash-users Google Groups in favor of a
> Discourse forum. You can read more about Discourse at
> http://www.discourse.org
>
> We feel Discourse will allow us to provide a better experience for all of
> our users for a few reasons:
>
> * More fine grained conversation topics = less noise and better targeted
> discussions. e.g. we can offer a forum for each language client, individual
> logstash plugin or for each city to plan user group meetings, etc.
>
> * Facilitates discussions that are not generally happening on list now,
> such as best practices by use case or tips from moving to development to
> production
>
> * Easier for folks who are purely end users - and less used to getting
> peer support on a mailing list - to get help when they need it
>
> Obviously, Discourse does not function the exact same way as a mailing
> list - however, email interaction with Discourse is supported and will
> continue to allow you to participate in discussions over email (though
> there are some small issues related to in-line replies. [0])
>
> We’re working with the Discourse team now as part of evaluati

Re[2]: Issue removing index with index.blocks.read_only set to true

2015-04-02 Thread nfraison via elasticsearch



Hi,
If I remuer well the isssue of setting back the read-only setting to false was 
due to the fact that I was performing a Put instead of a post or the opposite. 
Let me know if it help you, if no I will look at some old scripts that I should 
have kept with this.
Nicolas
jeudi, 02 avril 2015, 03:55PM +02:00 de BradVido :
>I have the same problem. You can't change the value to false because metadata 
>udpates are blocked and changing the value is itself a metadata update!
>Is there any way to delete this index now!?
>On Thursday, January 29, 2015 at 3:46:22 PM UTC-6, Mark Walkom wrote:
>>Just set that value to false and you should be good to go.
>>
>>On 29 January 2015 at 23:58, 'Nicolas Fraison' via elasticsearch  < 
>>elasti...@googlegroups.com > wrote:
>>>Hi,
>>>
>>>I was trying the settings to block data writes and also metadat writes to an 
>>>index applying this:
>>>curl -XPUT
>' http://testserver:9200/test_index/_settings ' -d ' {
>>>
>"index.blocks.read_only" : "true"
>>>}'
>>>
>>>which works fine but now I would like to remove this index and I'm facing 
>>>this issue;
>>>{"error":"RemoteTransportException[[lonsl1101844-master][inet[/10.243.1.10:9300]][indices:admin/delete]];
>nested: ClusterBlockException[blocked by: [FORBIDDEN/5/index read-only
>(api)];]; ","status":403}
>>>I can't change anymore the configuration of the index as metdata writes are 
>>>forbidden.
>>>
>>>I also now have an issues with my different monitoring tools as they are 
>>>using get _settings/_aliases.. request which are forbidden for this index.
>>>Could you please indicate me how I can force the remove of this index or 
>>>change its configuration?
>>>Thanks,
>>>Nicolas
>>>--
>>>You received this message because you are subscribed to the Google Groups 
>>>"elasticsearch" group.
>>>To unsubscribe from this group and stop receiving emails from it, send an 
>>>email to  elasticsearc...@googlegroups.com .
>>>To view this discussion on the web visit  
>>>https://groups.google.com/d/msgid/elasticsearch/06aaed45-0208-4e4f-8c3e-d58adf93a388%40googlegroups.com
>>> .
>>>For more options, visit  https://groups.google.com/d/optout .
>>
>--
>You received this message because you are subscribed to a topic in the Google 
>Groups "elasticsearch" group.
>To unsubscribe from this topic, visit  
>https://groups.google.com/d/topic/elasticsearch/JkGihq3zcQw/unsubscribe .
>To unsubscribe from this group and all its topics, send an email to  
>elasticsearch+unsubscr...@googlegroups.com .
>To view this discussion on the web visit  
>https://groups.google.com/d/msgid/elasticsearch/5f5f576a-bf33-44af-b671-e616b40e91af%40googlegroups.com
> .
>For more options, visit  https://groups.google.com/d/optout .

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1427989358.488102988%40f12.my.com.
For more options, visit https://groups.google.com/d/optout.

Evaluating Moving to Discourse - Feedback Wanted

2015-04-02 Thread leslie.hawthorn

Hello everyone,

As we’ve begun to scale up development on three different open source
projects, we’ve found Google Groups to be a difficult solution for dealing
with all of our needs for community support. We’ve got multiple mailing
lists going, which can be confusing for new folks trying to figure out
where to go to ask a question.

We’ve also found our lists are becoming noisy in the “good problem to have”
kind of way. As we’ve seen more user adoption, and across such a wide
variety of use cases, we’re getting widely different types of questions
asked. For example, I can imagine that folks not using our Python client
would rather not be distracted with emails about it.

There’s also a few other strikes against Groups as a tool, such as the fact
that it is no longer a supported product by Google, it provides no API
hooks and it is not available for users in China.

We’ve evaluated several options and we’re currently considering shuttering
the elasticsearch-user and logstash-users Google Groups in favor of a
Discourse forum. You can read more about Discourse at
http://www.discourse.org

We feel Discourse will allow us to provide a better experience for all of
our users for a few reasons:

* More fine grained conversation topics = less noise and better targeted
discussions. e.g. we can offer a forum for each language client, individual
logstash plugin or for each city to plan user group meetings, etc.

* Facilitates discussions that are not generally happening on list now,
such as best practices by use case or tips from moving to development to
production

* Easier for folks who are purely end users - and less used to getting peer
support on a mailing list - to get help when they need it

Obviously, Discourse does not function the exact same way as a mailing list
- however, email interaction with Discourse is supported and will continue
to allow you to participate in discussions over email (though there are
some small issues related to in-line replies. [0])

We’re working with the Discourse team now as part of evaluating this
transition, and we know they’re working to resolve this particular issue.
We’re also still determining how Discourse will handle our needs for both
user and list archive migration, and we’ll know the precise details of how
that would work soon. (We’ll share when we have them.)

The final goal would be to move Google Groups to read-only archives, and
cut over to Discourse completely for community support discussions.

We’re looking at making the cut over in ~30 days from today, but obviously
that’s subject to the feedback we receive from all of you. We’re sharing
this information to set expectations about time frame for making the
switch. It’s not set in stone. Our highest priority is to ensure effective
migration of our list archives and subscribers, which may mean a longer
time horizon for deploying Discourse, as well.

In the meantime, though, we wanted to communicate early and often and get
your feedback. Would this change make your life better? Worse? Meh?

Please share your thoughts with us so we can evaluate your feedback. We
don’t take this switch lightly, and we want to understand how it will
impact your overall workflow and experience.

We’ll make regular updates to the list responding to incoming feedback and
be completely transparent about how our thought processes evolve based on
it.

Thanks in advance!

[0] - https://meta.discourse.org/t/migrating-from-google-groups/24695

Cheers,
LH

Leslie Hawthorn
Director of Developer Relations
http://elastic.co

Other Places to Find Me:
Freenode: lh
Twitter: @lhawthorn

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a0c02cd4-231d-4ce5-adb7-7164de76a902%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Stop words returning results?

2015-04-02 Thread Rupert Smith

Sweet, thanks for your help.

On Thursday, April 2, 2015 at 3:16:29 PM UTC+1, Jörg Prante wrote:
>
> It is technically possible to combine analyzers for a single field, see 
> the combo analyzer https://github.com/yakaz/elasticsearch-analysis-combo/
>
> Jörg
>
> On Thu, Apr 2, 2015 at 11:12 AM, Rupert Smith  > wrote:
>
>> Ok thanks.
>>
>> Some of the fields are not_analyzed, and potentially some new fields 
>> could have different analyzers. I guess what you are saying is that if I 
>> want to use _all, I have to set it up with just one analyzer for all fields 
>> included in it? I can't mix say english and not_analyzed?
>>
>> I guess this is not such a problem really, since I don't yet have a use 
>> case for needing to query against multiple analyzers.
>>
>> Rupert
>>
>> On Thursday, April 2, 2015 at 10:09:21 AM UTC+1, Jörg Prante wrote:
>>>
>>> _all has its own analyzer, if you do not set it, it will be the standard 
>>> analyzer by default.
>>>
>>> Jörg
>>>
>>> On Thu, Apr 2, 2015 at 11:04 AM, Rupert Smith  
>>> wrote:
>>>
 Hi,

 I think I need to understand how the "_all" field works when it comes 
 to analysis. I want to query against all field that are indexed, not just 
 specific ones, this is for a generic free text search against content.

 I have created a stopword filter with just "the" and "and" in it. 

 I can query how the fields are analyzed and find that "the" is indeed 
 removed:

 curl http://localhost:9200/content/_analyze?field=content -d "the"
 {"tokens":[]}

 But if I check how the _all field is analyzed:

 curl http://localhost:9200/content/_analyze?field=_all -d "the"
 {"tokens":[{"token":"the","start_offset":0,"end_offset":
 3,"type":"","position":1}]}

 So when I query _all I get results that I don't want.

 Here is the start of my mappings, as you can see I did not set anything 
 on _all.

 {"content":{"mappings":{"business_rates_calculator":{
   "dynamic":"false",
   "properties":{
 "businessTypes":{"type":"string","index":"not_analyzed"},
 "content":{"type":"string","analyzer":"custom_english"},
 ...

 How do I set up _all so that it is composed of all the fields that are 
 mapped, each analyzed by the analyzer configured on the field. I assumed 
 this is how _all would work by default, but clearly not. 

 Thanks for your assistance.

 Rupert

 -- 
 You received this message because you are subscribed to the Google 
 Groups "elasticsearch" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to elasticsearc...@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/
 msgid/elasticsearch/0cee71aa-891b-4f11-9e3d-8510ac34a5fe%
 40googlegroups.com 
 
 .
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/760a2197-6f97-4ff9-956d-6dcf91b2a115%40googlegroups.com
>>  
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/3ae8b76d-c290-43a3-a1f3-67e750daeb3d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Problem with ES- Shield

2015-04-02 Thread 'BEN SALEM Omar' via elasticsearch

Hi Jay,
Now that I have shield in both ES and Kibana, and I want to load some data
from my hadoop cluster to ES.
I usually do this :


create external table es_word_count(word string, count int)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.resource' = 'test/test',
'es.nodes'='192.168.238.130:9200');

INSERT OVERWRITE TABLE es_word_count SELECT s.word, s.count  FROM
word_counts1 s;


what should I do or add to access to my ES cluster as the esuser I've
created?
Thanks
ᐧ

On Thu, Apr 2, 2015 at 1:08 PM, Jay Modi  wrote:

> Hi Omar,
>
> Have you configured Kibana and Marvel to work with Shield? For Kibana 4,
> please see
> http://www.elastic.co/guide/en/kibana/current/production.html#configuring-kibana-shield
> and
> http://www.elastic.co/guide/en/shield/current/_shield_with_kibana_4.html.
> For Marvel, see http://www.elastic.co/guide/en/shield/current/marvel.html
>
> -Jay
>
> On Thursday, April 2, 2015 at 6:45:55 AM UTC-4, BEN SALEM Omar wrote:
>>
>> Hi Jay,
>>
>> I have a question, now that I've added an esuser as an admin, I can not
>> run my Kibana anymore,
>>
>>
>> Any idea on what to do to overcome this?
>>
>> Another basic question, when I open the marvel, it shows no indice
>> because the sample of data I'm working with is in "2014", how to change my
>> time span to see my indices again?
>> btwn, before I install the shield, Marvel used to recognize my timestamp
>> by default
>>
>> 
>> ᐧ
>> Thank you,
>> Omar,
>>
>> On Wed, Apr 1, 2015 at 8:22 PM, BEN SALEM Omar 
>> wrote:
>>
>>> When I try to add an esuser and grant him a role:
>>>
>>> bin/shield/esusers useradd rdeniro -p taxidriver -r admin
>>>
>>> It says that there is an error in the "esusers" file in the
>>> elef['$ES_INCLUDE"] line, I think the line 31 or 38, I forgot.
>>> In the line above, there is a $...done: which ends with ":", if you
>>> change it to done; with ";" , the error doesn't show up anymore.
>>>
>>> I'm installing the whole thing in the Hortonworks Sandbox 2.0 vm
>>>
>>> On Wed, Apr 1, 2015 at 8:14 PM, Jay Modi  wrote:
>>>
 I'm glad you were able to fix this issue. I have a few questions that
 may help us ensure this fix is pulled back into Shield. What OS and Shell
 are you using? Which line in the script did you change exactly? I'm looking
 at the script right now and I don't see any instance of "done:"

 Just an FYI, there is another way to handle the configuration directory
 issue that is documented in the configuring your environment section of the
 shield guide:
 http://www.elastic.co/guide/en/shield/current/getting-
 started.html#_configuring_your_environment

 On Wednesday, April 1, 2015 at 2:07:28 PM UTC-4, BEN SALEM Omar wrote:
>
> Ok, I've managed to solve my own problem.
>
> First of all things, if you want to just remove the shield plugin run
> under the elasticsearch :
>
> $ bin/plugin --remove shield
>
> Now, to the syntax error in the elef line in the "esusers", generally,
> when there is such an error, it's related to things before that line.
> In fact there was something like this :
> $done:
> correction :
> $done;
>
> Now, we could add users within shield and grunt them the desired role.
>
> But, even if I've done it, I couldn't access to my ES cluster with the
> "new born user"; I think because my ES is running as a service and not
> starting with bin/elasticsearch, that lead to "misunderstanding."
>
> The solution I found was to transfer the /usr/share/elasticsearch/
> config/shield under /etc/elasticsearch
> $ln -s /usr/share/elasticsearch/config/shield/ /etc/elasticsearch
>
> And that was it !
> Hope that could be of a help for anyone else facing the same problem !
>
> On Wednesday, April 1, 2015 at 5:28:14 PM UTC+2, BEN SALEM Omar wrote:
>>
>> I wanted to add authentication to my ES, thus I downloaded the shield
>> :
>> https://www.elastic.co/downloads/shield
>>
>> Then I've tried to add a user to give him the admin role as it's
>> mentionned here :
>> http://www.elastic.co/guide/en/shield/current/getting-started.html
>> (I've added it with root, cause I start my es with root..)
>>
>> But when I try to do so, an error occure saying that there is a
>> syntax error in the 38 line "the elef line"
>> elef[$ES_INCLUDE] bla bla
>> I've tried many time :
>> bin/shield/esusers useradd omar -p omar -r admin
>> bin/shield/esusers useradd esomar -p omar -r admin
>> bin/shield/esusers useradd es_omar -p omar -r admin
>> bin/shield/esusers useradd ES_omar -p omar -r admin
>> bin/shield/esusers useradd omar  -r admin
>> bin/shield/esusers useradd esomar  -r admin
>> bin/shield/esusers useradd es_omar  -r admin
>> bin/shield/esusers useradd ES_omar  -r admin
>>
>> But the same error kept on jumping !
>>
>> I

Re: Stop words returning results?

It is technically possible to combine analyzers for a single field, see the
combo analyzer https://github.com/yakaz/elasticsearch-analysis-combo/

Jörg

On Thu, Apr 2, 2015 at 11:12 AM, Rupert Smith 
wrote:

> Ok thanks.
>
> Some of the fields are not_analyzed, and potentially some new fields could
> have different analyzers. I guess what you are saying is that if I want to
> use _all, I have to set it up with just one analyzer for all fields
> included in it? I can't mix say english and not_analyzed?
>
> I guess this is not such a problem really, since I don't yet have a use
> case for needing to query against multiple analyzers.
>
> Rupert
>
> On Thursday, April 2, 2015 at 10:09:21 AM UTC+1, Jörg Prante wrote:
>>
>> _all has its own analyzer, if you do not set it, it will be the standard
>> analyzer by default.
>>
>> Jörg
>>
>> On Thu, Apr 2, 2015 at 11:04 AM, Rupert Smith 
>> wrote:
>>
>>> Hi,
>>>
>>> I think I need to understand how the "_all" field works when it comes to
>>> analysis. I want to query against all field that are indexed, not just
>>> specific ones, this is for a generic free text search against content.
>>>
>>> I have created a stopword filter with just "the" and "and" in it.
>>>
>>> I can query how the fields are analyzed and find that "the" is indeed
>>> removed:
>>>
>>> curl http://localhost:9200/content/_analyze?field=content -d "the"
>>> {"tokens":[]}
>>>
>>> But if I check how the _all field is analyzed:
>>>
>>> curl http://localhost:9200/content/_analyze?field=_all -d "the"
>>> {"tokens":[{"token":"the","start_offset":0,"end_offset":
>>> 3,"type":"","position":1}]}
>>>
>>> So when I query _all I get results that I don't want.
>>>
>>> Here is the start of my mappings, as you can see I did not set anything
>>> on _all.
>>>
>>> {"content":{"mappings":{"business_rates_calculator":{
>>>   "dynamic":"false",
>>>   "properties":{
>>> "businessTypes":{"type":"string","index":"not_analyzed"},
>>> "content":{"type":"string","analyzer":"custom_english"},
>>> ...
>>>
>>> How do I set up _all so that it is composed of all the fields that are
>>> mapped, each analyzed by the analyzer configured on the field. I assumed
>>> this is how _all would work by default, but clearly not.
>>>
>>> Thanks for your assistance.
>>>
>>> Rupert
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/elasticsearch/0cee71aa-891b-4f11-9e3d-8510ac34a5fe%
>>> 40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/760a2197-6f97-4ff9-956d-6dcf91b2a115%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEp9cD1qi0xs1L%2BvEda8MFL2_weR6f-y%2BoORVoRKQ9X7g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

It seems impossible to modify/delete an index after setting blocks.read_only=true

2015-04-02 Thread BradVido

I have an index, let's call it myindex, and I updated the settings to make 
it read-only like so:
PUT myindex/_settings  {"blocks": { "read_only": "true" }}
This worked as expected, but now I am stuck! I need to delete this index, 
but I cannot modify this index because blocks.read_only also disables 
meta-data updates.

I tried to do undo it like so, but it failed:

PUT myindex/_settings  {"blocks": { "read_only": "false" }}

ClusterBlockException[blocked by: [FORBIDDEN/5/index read-only (api)];];


So I thought closing it first would help, but that failed too:

POST myindex/_close

ClusterBlockException[blocked by: [FORBIDDEN/5/index read-only (api)];];

Deleting the index failed with the same exception.

Am I stuck with this index until then end of time!?!

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/aefdc538-3a01-4894-8bd8-da036c1aedd4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: transport client threadpool and down node

Yes, that is the purpose of "sniff". You do not need to add all known nodes
to the TransportClient, it will autodiscover other nodes when "sniff" is
enabled.

Jörg

On Thu, Apr 2, 2015 at 1:46 PM, Jason Wee  wrote:

> yes, I have added all known nodes in the cluster into the transport
> client side.. so does this sniff parameter added will also work when
> the downed node come back online?
>
> jason
>
> On Thu, Apr 2, 2015 at 7:25 PM, joergpra...@gmail.com
>  wrote:
> > You can set "client.transport.sniff" to true, then nodes are detected
> > automatically
> >
> >
> http://www.elastic.co/guide/en/elasticsearch/client/java-api/current/client.html#transport-client
> >
> > Jörg
> >
> > On Thu, Apr 2, 2015 at 12:25 PM, Jason Wee  wrote:
> >>
> >> Hello, elasticsearch java transport client , for index, is the thread
> >> spawn bounded? if not, is there a way to set it to a fix value? When
> one of
> >> a node in cluster is down, java transport client will remove this downed
> >> node from its list, right? But when the downed node has joined back
> into the
> >> cluster, does the transport client automatically add the new node back
> to
> >> its list?
> >>
> >> Thanks.
> >>
> >> Jason
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> Groups
> >> "elasticsearch" group.
> >> To unsubscribe from this group and stop receiving emails from it, send
> an
> >> email to elasticsearch+unsubscr...@googlegroups.com.
> >> To view this discussion on the web visit
> >>
> https://groups.google.com/d/msgid/elasticsearch/639ada4d-1176-4e2a-b437-a81f8ec72d4f%40googlegroups.com
> .
> >> For more options, visit https://groups.google.com/d/optout.
> >
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to elasticsearch+unsubscr...@googlegroups.com.
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG8CP09KftpumV29BnTsx%2BjJ6ttK4BzM3ZSvHv07B7wkQ%40mail.gmail.com
> .
> >
> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAHO4itzWC4YJm7Km3vbaXcSPvPMbzXc45PzugOoY4fD3Jo9ukg%40mail.gmail.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEmmy2MXQ5HDUCMZqRR-hc8_StUT3Dw5p3RGfcEEMhZdg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Issue removing index with index.blocks.read_only set to true

2015-04-02 Thread BradVido

I have the same problem. You can't change the value to false because 
metadata udpates are blocked and changing the value is itself a metadata 
update!
Is there any way to delete this index now!? 

On Thursday, January 29, 2015 at 3:46:22 PM UTC-6, Mark Walkom wrote:
>
> Just set that value to false and you should be good to go.
>
> On 29 January 2015 at 23:58, 'Nicolas Fraison' via elasticsearch <
> elasti...@googlegroups.com > wrote:
>
>> Hi,
>>
>> I was trying the settings to block data writes and also metadat writes to 
>> an index applying this:
>>
>> curl -XPUT 'http://testserver:9200/test_index/_settings' -d '{
>> "index.blocks.read_only" : "true"
>> }' 
>>
>>
>> which works fine but now I would like to remove this index and I'm facing 
>> this issue;
>>
>> *{"error":"RemoteTransportException[[lonsl1101844-master][inet[/10.243.1.10:9300]][indices:admin/delete]];
>>  
>> nested: ClusterBlockException[blocked by: [FORBIDDEN/5/index read-only 
>> (api)];]; ","status":403}*
>>
>> I can't change anymore the configuration of the index as metdata writes 
>> are forbidden.
>>
>>
>> I also now have an issues with my different monitoring tools as they are 
>> using get _settings/_aliases.. request which are forbidden for this index.
>>
>> Could you please indicate me how I can force the remove of this index or 
>> change its configuration?
>>
>> Thanks,
>> Nicolas
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/06aaed45-0208-4e4f-8c3e-d58adf93a388%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5f5f576a-bf33-44af-b671-e616b40e91af%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ES_HEAP_SIZE Size with 10k Doc's (5 MB total)?

2015-04-02 Thread David Pilato

Indeed. Start small 1gb or 2 and monitor your usage.
Increase if you see your HEAP usage going up to 75%/80%.

My 2 cents.

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

> Le 2 avr. 2015 à 15:28, Arno Nühm  a écrit :
> 
> I've read a lot of ES Performance Guides and most of them suggest setting the 
> ES_HEAP_SIZE to half the RAM, but not more than 32 GB.
> 
> ES is running on a Server with 32 GB of RAM, with nginx, php5-fpm and mysql 
> on this machine and i'm only indexing around 10.000 Documents with a total 
> size of 5 MB. 
> The Index will probably grow to around 20.000 - 30.000 (10 - 15 MB) Documents 
> in the near future, but thats - compared to other clusters - is still a 
> fairly small amount of data.
> 
> I dont have heavy traffic nor i'm doing crazy searches. 
> Most types are not_analyzed, because i'm primarily using ES for filtering, 
> aggregations and sorting, full text search is just a minor benefit for me, 
> but it would be crazy doing all the filtering with mysql and not with ES.
> 
> Setting the ES_HEAP_SIZE to around 16 GB seems a bit of an overkill for my 
> usecase, but since i'm using filters (which are cacheable) and doing 
> aggregations with sorting, setting the ES_HEAP_SIZE at around 2 GB feels 
> kinda right for me, but i'm not sure if i'm missing something here.
> 
> Any reccommendation on the ES_HEAP_SIZE with a small number of documents is 
> really appreciated, thanks!
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/f4a5b85b-d50e-4a29-837e-945c21fbd99f%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/3C2D6FEA-FF94-446B-9F35-A913C2C3F8DA%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

ES_HEAP_SIZE Size with 10k Doc's (5 MB total)?

2015-04-02 Thread Arno Nühm

I've read a lot of ES Performance Guides and most of them suggest setting
the ES_HEAP_SIZE to half the RAM, but not more than 32 GB.

ES is running on a Server with 32 GB of RAM, with nginx, php5-fpm and mysql
on this machine and i'm only indexing around 10.000 Documents with a total
size of 5 MB.

The Index will probably grow to around 20.000 - 30.000 (10 - 15 MB)
Documents in the near future, but thats - compared to other clusters - is
still a fairly small amount of data.

I dont have heavy traffic nor i'm doing crazy searches.

Most types are not_analyzed, because i'm primarily using ES for filtering,
aggregations and sorting, full text search is just a minor benefit for me,
but it would be crazy doing all the filtering with mysql and not with ES.

Setting the ES_HEAP_SIZE to around 16 GB seems a bit of an overkill for my
usecase, but since i'm using filters (which are cacheable) and doing
aggregations with sorting, setting the ES_HEAP_SIZE at around 2 GB feels
kinda right for me, but i'm not sure if i'm missing something here.

Any reccommendation on the ES_HEAP_SIZE with a small number of documents is
really appreciated, thanks!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f4a5b85b-d50e-4a29-837e-945c21fbd99f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

UNASSIGNED indexes

2015-04-02 Thread Darius

Hello,

Quick description of the problem: I had 1 elasticsearch server cluster with
0 replicas for all indexes, then added a second ES server to the cluster
and then set that all existing indexes should have 1 replica (instead of
the 0) .
After doing this, in the _cat/shards page I could see that it created twice
as many indexes/indexes(?) (sorry, im not that good with the ES
terminology) and all the new ones were UNASSIGNED. They then started slowly
INITIALIZING (2 at a time) and then go to STARTED state on the second
server. So thats good, but the problem is that this was done like 2 weeks
ago and some of them are still UNASSIGNED or INITIALIZING. I've even
noticed some of them going into the INITIALIZING state and then falling
back to UNASSIGNED. How can I properly find the problem here and is there
any way I can force the replicas to be allocated to the 2nd node WITHOUT
any data loss? I found only one way to allocate, but that includes data
loss, unfortunately.

I don't have tons of data in there so I dont see why it should take so
long. The largest index is ±500M in size, most of them are a few megs and
the total number is around ±150-200.

any help would be appreciated..

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/05cad3e5-78ce-44f7-9a30-6ca0eb817237%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ElasticSearch Multi Data Center replication

2015-04-02 Thread Vladi Feigin

Thank you Itamar,
Can you describe the process with the more details?
Is it something like that:

1. Extract a snapshot in cluster_source with snapshot API
2. Copy it to target cluster
3. Restore it in the target cluster with restore API

All this wrap in the two daemon jobs , one for snapshot (running in the
source cluster) and other for restoring running in the target cluster?
Thank you,
Vladi


On Thu, Apr 2, 2015 at 2:58 PM, Itamar Syn-Hershko 
wrote:

> Use the snapshot / restore API, unless you care for latency then look at
> implementing a daemon tool to do that. You should try to keep them decoupled
>
> --
>
> Itamar Syn-Hershko
> http://code972.com | @synhershko 
> Freelance Developer & Consultant
> Lucene.NET committer and PMC member
>
> On Thu, Apr 2, 2015 at 2:48 PM, Vladi Feigin 
> wrote:
>
>> Hi,
>>
>> Is there a solution for this ? Could you recommend some production proven?
>> Thank you
>> Vladi
>>
>> This message may contain confidential and/or privileged information.
>> If you are not the addressee or authorized to receive this on behalf of
>> the addressee you must not use, copy, disclose or take action based on this
>> message or any information herein.
>> If you have received this message in error, please advise the sender
>> immediately by reply email and delete this message. Thank you.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/bb6ef682-5a94-4f2e-9cfb-345a9c31ffa7%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/iRMQb8vVG_w/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtAMWcLLtJP0WTgFkofBBa04wW%2Buypx456BnQYLmVwmvw%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
This message may contain confidential and/or privileged information. 
If you are not the addressee or authorized to receive this on behalf of the 
addressee you must not use, copy, disclose or take action based on this 
message or any information herein. 
If you have received this message in error, please advise the sender 
immediately by reply email and delete this message. Thank you.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CACvWdiqR%2B9trJZ6gv2fbf0FiyEsL_C%2BazrE7D4V0L7aw3T%3D_Sg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: ElasticSearch Multi Data Center replication

2015-04-02 Thread Itamar Syn-Hershko

Use the snapshot / restore API, unless you care for latency then look at
implementing a daemon tool to do that. You should try to keep them decoupled

--

Itamar Syn-Hershko
http://code972.com | @synhershko 
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Thu, Apr 2, 2015 at 2:48 PM, Vladi Feigin  wrote:

> Hi,
>
> Is there a solution for this ? Could you recommend some production proven?
> Thank you
> Vladi
>
> This message may contain confidential and/or privileged information.
> If you are not the addressee or authorized to receive this on behalf of
> the addressee you must not use, copy, disclose or take action based on this
> message or any information herein.
> If you have received this message in error, please advise the sender
> immediately by reply email and delete this message. Thank you.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/bb6ef682-5a94-4f2e-9cfb-345a9c31ffa7%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtAMWcLLtJP0WTgFkofBBa04wW%2Buypx456BnQYLmVwmvw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

ElasticSearch Multi Data Center replication

2015-04-02 Thread Vladi Feigin

Hi,

Is there a solution for this ? Could you recommend some production proven?
Thank you
Vladi

-- 
This message may contain confidential and/or privileged information. 
If you are not the addressee or authorized to receive this on behalf of the 
addressee you must not use, copy, disclose or take action based on this 
message or any information herein. 
If you have received this message in error, please advise the sender 
immediately by reply email and delete this message. Thank you.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/bb6ef682-5a94-4f2e-9cfb-345a9c31ffa7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: transport client threadpool and down node

2015-04-02 Thread Jason Wee

yes, I have added all known nodes in the cluster into the transport
client side.. so does this sniff parameter added will also work when
the downed node come back online?

jason

On Thu, Apr 2, 2015 at 7:25 PM, joergpra...@gmail.com
 wrote:
> You can set "client.transport.sniff" to true, then nodes are detected
> automatically
>
> http://www.elastic.co/guide/en/elasticsearch/client/java-api/current/client.html#transport-client
>
> Jörg
>
> On Thu, Apr 2, 2015 at 12:25 PM, Jason Wee  wrote:
>>
>> Hello, elasticsearch java transport client , for index, is the thread
>> spawn bounded? if not, is there a way to set it to a fix value? When one of
>> a node in cluster is down, java transport client will remove this downed
>> node from its list, right? But when the downed node has joined back into the
>> cluster, does the transport client automatically add the new node back to
>> its list?
>>
>> Thanks.
>>
>> Jason
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/639ada4d-1176-4e2a-b437-a81f8ec72d4f%40googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG8CP09KftpumV29BnTsx%2BjJ6ttK4BzM3ZSvHv07B7wkQ%40mail.gmail.com.
>
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAHO4itzWC4YJm7Km3vbaXcSPvPMbzXc45PzugOoY4fD3Jo9ukg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Expand sharding on exist index

ThaT
Thank you for the response , 
regarding second option, i want to reindex current index, as the 
performance is already very slow and we can leave it as is.
the problem is that we have types that not stored ( "_source" : { "enabled" 
: false } ) how can we reindex such doc types without retrieving them ? is 
there any special api for this purpose ?



On Thursday, April 2, 2015 at 11:31:23 AM UTC+3, Grigory Rubstein wrote:
>
> Hi
>
> Is its possible to change number of shards on exist index, where part of 
> the types does not stored fields ?
>
> This message may contain confidential and/or privileged information. 
> If you are not the addressee or authorized to receive this on behalf of 
> the addressee you must not use, copy, disclose or take action based on this 
> message or any information herein. 
> If you have received this message in error, please advise the sender 
> immediately by reply email and delete this message. Thank you.
>

-- 
This message may contain confidential and/or privileged information. 
If you are not the addressee or authorized to receive this on behalf of the 
addressee you must not use, copy, disclose or take action based on this 
message or any information herein. 
If you have received this message in error, please advise the sender 
immediately by reply email and delete this message. Thank you.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/475226eb-fb9d-4f8a-ae60-f71ac58670ea%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: transport client threadpool and down node

You can set "client.transport.sniff" to true, then nodes are detected
automatically

http://www.elastic.co/guide/en/elasticsearch/client/java-api/current/client.html#transport-client

Jörg

On Thu, Apr 2, 2015 at 12:25 PM, Jason Wee  wrote:

> Hello, elasticsearch java transport client , for index, is the thread
> spawn bounded? if not, is there a way to set it to a fix value? When one of
> a node in cluster is down, java transport client will remove this downed
> node from its list, right? But when the downed node has joined back into
> the cluster, does the transport client automatically add the new node back
> to its list?
>
> Thanks.
>
> Jason
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/639ada4d-1176-4e2a-b437-a81f8ec72d4f%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG8CP09KftpumV29BnTsx%2BjJ6ttK4BzM3ZSvHv07B7wkQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: geo_bounding_box returns 0 results every time

2015-04-02 Thread Marius Miliunas

Duh... Thanks a ton Christian. I was starting to go nuts.

On Thursday, April 2, 2015 at 11:42:18 AM UTC+2, chri...@elastic.co wrote:
>
> Hi,
>
> Should the top_left latitude not be higher than the bottom_right latitude, 
> giving the following bounding box?
>
> "geo_bounding_box":{  
> "coords":{  
> "top_left":{  
> "lat":55,
> "lon":10
> },
> "bottom_right":{  
> "lat":50,
> "lon":15
> }
> }
> }
>
> Best regards,
>
> Christian
>
>
>
>
>
> On Thursday, April 2, 2015 at 10:09:26 AM UTC+1, Marius Miliunas wrote:
>>
>> Am experiencing a geo bounding box issue that returns 0 results no matter 
>> how I modify the query
>>
>> *This gist * contains 
>> the setup and query along with a similar query that should work.
>>
>> I've tested on ES 1.3.0 and 1.5.0
>>
>> Am I missing something here?
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/450f0e6a-2bc6-4b2a-a5ea-a7396c4903af%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: geo_bounding_box returns 0 results every time

2015-04-02 Thread Marius Miliunas

Duh... Thanks a ton Chris. I was starting to go nuts.

On Thursday, April 2, 2015 at 11:09:26 AM UTC+2, Marius Miliunas wrote:
>
> Am experiencing a geo bounding box issue that returns 0 results no matter 
> how I modify the query
>
> *This gist * contains 
> the setup and query along with a similar query that should work.
>
> I've tested on ES 1.3.0 and 1.5.0
>
> Am I missing something here?
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/df548ca2-92cb-43d7-bda4-91ed090ecc24%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Problem with ES- Shield

Hi Omar,

Have you configured Kibana and Marvel to work with Shield? For Kibana 4, 
please 
see 
http://www.elastic.co/guide/en/kibana/current/production.html#configuring-kibana-shield
 
and http://www.elastic.co/guide/en/shield/current/_shield_with_kibana_4.html. 
For Marvel, see http://www.elastic.co/guide/en/shield/current/marvel.html

-Jay

On Thursday, April 2, 2015 at 6:45:55 AM UTC-4, BEN SALEM Omar wrote:
>
> Hi Jay,
>
> I have a question, now that I've added an esuser as an admin, I can not 
> run my Kibana anymore, 
>
>
> Any idea on what to do to overcome this?
>
> Another basic question, when I open the marvel, it shows no indice because 
> the sample of data I'm working with is in "2014", how to change my time 
> span to see my indices again?
> btwn, before I install the shield, Marvel used to recognize my timestamp 
> by default
>
> 
> ᐧ
> Thank you,
> Omar,
>
> On Wed, Apr 1, 2015 at 8:22 PM, BEN SALEM Omar  > wrote:
>
>> When I try to add an esuser and grant him a role:
>>
>> bin/shield/esusers useradd rdeniro -p taxidriver -r admin
>>
>> It says that there is an error in the "esusers" file in the 
>> elef['$ES_INCLUDE"] line, I think the line 31 or 38, I forgot.
>> In the line above, there is a $...done: which ends with ":", if you 
>> change it to done; with ";" , the error doesn't show up anymore.
>>
>> I'm installing the whole thing in the Hortonworks Sandbox 2.0 vm
>>
>> On Wed, Apr 1, 2015 at 8:14 PM, Jay Modi > 
>> wrote:
>>
>>> I'm glad you were able to fix this issue. I have a few questions that 
>>> may help us ensure this fix is pulled back into Shield. What OS and Shell 
>>> are you using? Which line in the script did you change exactly? I'm looking 
>>> at the script right now and I don't see any instance of "done:"
>>>
>>> Just an FYI, there is another way to handle the configuration directory 
>>> issue that is documented in the configuring your environment section of the 
>>> shield guide:
>>>
>>> http://www.elastic.co/guide/en/shield/current/getting-started.html#_configuring_your_environment
>>>
>>> On Wednesday, April 1, 2015 at 2:07:28 PM UTC-4, BEN SALEM Omar wrote:

 Ok, I've managed to solve my own problem.

 First of all things, if you want to just remove the shield plugin run 
 under the elasticsearch :

 $ bin/plugin --remove shield

 Now, to the syntax error in the elef line in the "esusers", generally, 
 when there is such an error, it's related to things before that line.
 In fact there was something like this : 
 $done: 
 correction : 
 $done;

 Now, we could add users within shield and grunt them the desired role.

 But, even if I've done it, I couldn't access to my ES cluster with the 
 "new born user"; I think because my ES is running as a service and not 
 starting with bin/elasticsearch, that lead to "misunderstanding."

 The solution I found was to transfer the 
 /usr/share/elasticsearch/config/shield 
 under /etc/elasticsearch
 $ln -s /usr/share/elasticsearch/config/shield/ /etc/elasticsearch

 And that was it ! 
 Hope that could be of a help for anyone else facing the same problem !

 On Wednesday, April 1, 2015 at 5:28:14 PM UTC+2, BEN SALEM Omar wrote:
>
> I wanted to add authentication to my ES, thus I downloaded the shield :
> https://www.elastic.co/downloads/shield
>
> Then I've tried to add a user to give him the admin role as it's 
> mentionned here :
> http://www.elastic.co/guide/en/shield/current/getting-started.html
> (I've added it with root, cause I start my es with root..)
>
> But when I try to do so, an error occure saying that there is a syntax 
> error in the 38 line "the elef line" 
> elef[$ES_INCLUDE] bla bla
> I've tried many time : 
> bin/shield/esusers useradd omar -p omar -r admin
> bin/shield/esusers useradd esomar -p omar -r admin
> bin/shield/esusers useradd es_omar -p omar -r admin
> bin/shield/esusers useradd ES_omar -p omar -r admin
> bin/shield/esusers useradd omar  -r admin
> bin/shield/esusers useradd esomar  -r admin
> bin/shield/esusers useradd es_omar  -r admin
> bin/shield/esusers useradd ES_omar  -r admin
>
> But the same error kept on jumping ! 
>
> I said enough is enough and wanted to keep on doing wwhat I was doing, 
> tried to run Kibana, and this is what happened !
>
>
> I've then tried to open my ES marvel plugin, but there is now an 
> authentication box 
>
>
> 
> I've tried ALL the combinations above, but none of them seemed to work 
> !!
>
> I'm really preoccupied by this note : 
> To ensure that Elasticsearch can read the user and role information at 
> startup, run esusers useradd as the same user you use to run 
> Elasticsearch. Running the command as root or some other user will update 
> the permissions for the user

Re: Problem with ES- Shield

2015-04-02 Thread 'BEN SALEM Omar' via elasticsearch

Hi Jay,

I have a question, now that I've added an esuser as an admin, I can not run
my Kibana anymore,


Any idea on what to do to overcome this?

Another basic question, when I open the marvel, it shows no indice because
the sample of data I'm working with is in "2014", how to change my time
span to see my indices again?
btwn, before I install the shield, Marvel used to recognize my timestamp by
default


ᐧ
Thank you,
Omar,

On Wed, Apr 1, 2015 at 8:22 PM, BEN SALEM Omar 
wrote:

> When I try to add an esuser and grant him a role:
>
> bin/shield/esusers useradd rdeniro -p taxidriver -r admin
>
> It says that there is an error in the "esusers" file in the
> elef['$ES_INCLUDE"] line, I think the line 31 or 38, I forgot.
> In the line above, there is a $...done: which ends with ":", if you change
> it to done; with ";" , the error doesn't show up anymore.
>
> I'm installing the whole thing in the Hortonworks Sandbox 2.0 vm
>
> On Wed, Apr 1, 2015 at 8:14 PM, Jay Modi  wrote:
>
>> I'm glad you were able to fix this issue. I have a few questions that may
>> help us ensure this fix is pulled back into Shield. What OS and Shell are
>> you using? Which line in the script did you change exactly? I'm looking at
>> the script right now and I don't see any instance of "done:"
>>
>> Just an FYI, there is another way to handle the configuration directory
>> issue that is documented in the configuring your environment section of the
>> shield guide:
>>
>> http://www.elastic.co/guide/en/shield/current/getting-started.html#_configuring_your_environment
>>
>> On Wednesday, April 1, 2015 at 2:07:28 PM UTC-4, BEN SALEM Omar wrote:
>>>
>>> Ok, I've managed to solve my own problem.
>>>
>>> First of all things, if you want to just remove the shield plugin run
>>> under the elasticsearch :
>>>
>>> $ bin/plugin --remove shield
>>>
>>> Now, to the syntax error in the elef line in the "esusers", generally,
>>> when there is such an error, it's related to things before that line.
>>> In fact there was something like this :
>>> $done:
>>> correction :
>>> $done;
>>>
>>> Now, we could add users within shield and grunt them the desired role.
>>>
>>> But, even if I've done it, I couldn't access to my ES cluster with the
>>> "new born user"; I think because my ES is running as a service and not
>>> starting with bin/elasticsearch, that lead to "misunderstanding."
>>>
>>> The solution I found was to transfer the 
>>> /usr/share/elasticsearch/config/shield
>>> under /etc/elasticsearch
>>> $ln -s /usr/share/elasticsearch/config/shield/ /etc/elasticsearch
>>>
>>> And that was it !
>>> Hope that could be of a help for anyone else facing the same problem !
>>>
>>> On Wednesday, April 1, 2015 at 5:28:14 PM UTC+2, BEN SALEM Omar wrote:

 I wanted to add authentication to my ES, thus I downloaded the shield :
 https://www.elastic.co/downloads/shield

 Then I've tried to add a user to give him the admin role as it's
 mentionned here :
 http://www.elastic.co/guide/en/shield/current/getting-started.html
 (I've added it with root, cause I start my es with root..)

 But when I try to do so, an error occure saying that there is a syntax
 error in the 38 line "the elef line"
 elef[$ES_INCLUDE] bla bla
 I've tried many time :
 bin/shield/esusers useradd omar -p omar -r admin
 bin/shield/esusers useradd esomar -p omar -r admin
 bin/shield/esusers useradd es_omar -p omar -r admin
 bin/shield/esusers useradd ES_omar -p omar -r admin
 bin/shield/esusers useradd omar  -r admin
 bin/shield/esusers useradd esomar  -r admin
 bin/shield/esusers useradd es_omar  -r admin
 bin/shield/esusers useradd ES_omar  -r admin

 But the same error kept on jumping !

 I said enough is enough and wanted to keep on doing wwhat I was doing,
 tried to run Kibana, and this is what happened !


 I've then tried to open my ES marvel plugin, but there is now an
 authentication box


 
 I've tried ALL the combinations above, but none of them seemed to work
 !!

 I'm really preoccupied by this note :
 To ensure that Elasticsearch can read the user and role information at
 startup, run esusers useradd as the same user you use to run
 Elasticsearch. Running the command as root or some other user will update
 the permissions for the users and users_roles files and prevent
 Elasticsearch from accessing them.

 The thing is that I start my es and kibana with the root..
 Please HELP !!

>>>  --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "elasticsearch" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/elasticsearch/sEZpCO7E_B0/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.go

Elasticsearch Scan and Scroll

2015-04-02 Thread lakshay joshi

Hi All,

My question is related to Scan and Scroll API.
Can we include SCAN and SCROLL with in single elasticsearch query template 
to get all the results.

Thanks & Regards
Lakshay Joshi

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/7941d404-d52e-4491-8108-ff3bee3881e4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

transport client threadpool and down node

2015-04-02 Thread Jason Wee

Hello, elasticsearch java transport client , for index, is the thread spawn 
bounded? if not, is there a way to set it to a fix value? When one of a 
node in cluster is down, java transport client will remove this downed node 
from its list, right? But when the downed node has joined back into the 
cluster, does the transport client automatically add the new node back to 
its list?

Thanks.

Jason

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/639ada4d-1176-4e2a-b437-a81f8ec72d4f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: geo_bounding_box returns 0 results every time

2015-04-02 Thread christian

Hi,

Should the top_left latitude not be higher than the bottom_right latitude, 
giving the following bounding box?

"geo_bounding_box":{  
"coords":{  
"top_left":{  
"lat":55,
"lon":10
},
"bottom_right":{  
"lat":50,
"lon":15
}
}
}

Best regards,

Christian





On Thursday, April 2, 2015 at 10:09:26 AM UTC+1, Marius Miliunas wrote:
>
> Am experiencing a geo bounding box issue that returns 0 results no matter 
> how I modify the query
>
> *This gist * contains 
> the setup and query along with a similar query that should work.
>
> I've tested on ES 1.3.0 and 1.5.0
>
> Am I missing something here?
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/c2947d3e-120f-4e65-a852-9dfb9cd4d106%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Multiple sub aggregation in java api

[solved]

here is the solution:
AggregationBuilders.terms("by_gender").field("gende").size(10).

subAggregation(AggregationBuilders.terms("by_city").field("city")).

subAggregation(AggregationBuilders.terms("by_language").field("language"));



On Thursday, April 2, 2015 at 11:29:07 AM UTC+3, Grigory Rubstein wrote:
>
> Hi
>
> I wondering if the java client support multiple sub aggregation on same 
> level :
>
> for example how to translate this to java api :
> by_city and by_language is count aggregation on same level.
>
> {
>   "aggs": {
> "by_gender": {
>   "terms": {
> "field": "gender",
> "size": 10
>   },
>   "aggs": {
> "by_city": {
>   "terms": {
> "field": "city",
> "size": 3
>   }
> },
> "by_language": {
>   "terms": {
> "field": "gender",
> "size": 1
>   }
> }
>   }
> }
>   }
> }
>
> This message may contain confidential and/or privileged information. 
> If you are not the addressee or authorized to receive this on behalf of 
> the addressee you must not use, copy, disclose or take action based on this 
> message or any information herein. 
> If you have received this message in error, please advise the sender 
> immediately by reply email and delete this message. Thank you.
>

-- 
This message may contain confidential and/or privileged information. 
If you are not the addressee or authorized to receive this on behalf of the 
addressee you must not use, copy, disclose or take action based on this 
message or any information herein. 
If you have received this message in error, please advise the sender 
immediately by reply email and delete this message. Thank you.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d30428a3-72da-4460-85dc-a1bb205b0d9a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Stop words returning results?

2015-04-02 Thread Rupert Smith

Ok thanks.

Some of the fields are not_analyzed, and potentially some new fields could 
have different analyzers. I guess what you are saying is that if I want to 
use _all, I have to set it up with just one analyzer for all fields 
included in it? I can't mix say english and not_analyzed?

I guess this is not such a problem really, since I don't yet have a use 
case for needing to query against multiple analyzers.

Rupert

On Thursday, April 2, 2015 at 10:09:21 AM UTC+1, Jörg Prante wrote:
>
> _all has its own analyzer, if you do not set it, it will be the standard 
> analyzer by default.
>
> Jörg
>
> On Thu, Apr 2, 2015 at 11:04 AM, Rupert Smith  > wrote:
>
>> Hi,
>>
>> I think I need to understand how the "_all" field works when it comes to 
>> analysis. I want to query against all field that are indexed, not just 
>> specific ones, this is for a generic free text search against content.
>>
>> I have created a stopword filter with just "the" and "and" in it. 
>>
>> I can query how the fields are analyzed and find that "the" is indeed 
>> removed:
>>
>> curl http://localhost:9200/content/_analyze?field=content -d "the"
>> {"tokens":[]}
>>
>> But if I check how the _all field is analyzed:
>>
>> curl http://localhost:9200/content/_analyze?field=_all -d "the"
>>
>> {"tokens":[{"token":"the","start_offset":0,"end_offset":3,"type":"","position":1}]}
>>
>> So when I query _all I get results that I don't want.
>>
>> Here is the start of my mappings, as you can see I did not set anything 
>> on _all.
>>
>> {"content":{"mappings":{"business_rates_calculator":{
>>   "dynamic":"false",
>>   "properties":{
>> "businessTypes":{"type":"string","index":"not_analyzed"},
>> "content":{"type":"string","analyzer":"custom_english"},
>> ...
>>
>> How do I set up _all so that it is composed of all the fields that are 
>> mapped, each analyzed by the analyzer configured on the field. I assumed 
>> this is how _all would work by default, but clearly not. 
>>
>> Thanks for your assistance.
>>
>> Rupert
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/0cee71aa-891b-4f11-9e3d-8510ac34a5fe%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/760a2197-6f97-4ff9-956d-6dcf91b2a115%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

geo_bounding_box returns 0 results every time

2015-04-02 Thread Marius Miliunas



Am experiencing a geo bounding box issue that returns 0 results no matter 
how I modify the query

*This gist * contains 
the setup and query along with a similar query that should work.

I've tested on ES 1.3.0 and 1.5.0

Am I missing something here?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/a2b7abcd-c75b-4fd8-807e-cc25385f442c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Stop words returning results?

_all has its own analyzer, if you do not set it, it will be the standard
analyzer by default.

Jörg

On Thu, Apr 2, 2015 at 11:04 AM, Rupert Smith 
wrote:

> Hi,
>
> I think I need to understand how the "_all" field works when it comes to
> analysis. I want to query against all field that are indexed, not just
> specific ones, this is for a generic free text search against content.
>
> I have created a stopword filter with just "the" and "and" in it.
>
> I can query how the fields are analyzed and find that "the" is indeed
> removed:
>
> curl http://localhost:9200/content/_analyze?field=content -d "the"
> {"tokens":[]}
>
> But if I check how the _all field is analyzed:
>
> curl http://localhost:9200/content/_analyze?field=_all -d "the"
>
> {"tokens":[{"token":"the","start_offset":0,"end_offset":3,"type":"","position":1}]}
>
> So when I query _all I get results that I don't want.
>
> Here is the start of my mappings, as you can see I did not set anything on
> _all.
>
> {"content":{"mappings":{"business_rates_calculator":{
>   "dynamic":"false",
>   "properties":{
> "businessTypes":{"type":"string","index":"not_analyzed"},
> "content":{"type":"string","analyzer":"custom_english"},
> ...
>
> How do I set up _all so that it is composed of all the fields that are
> mapped, each analyzed by the analyzer configured on the field. I assumed
> this is how _all would work by default, but clearly not.
>
> Thanks for your assistance.
>
> Rupert
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/0cee71aa-891b-4f11-9e3d-8510ac34a5fe%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFpiVq75c%2B11aUir8YovZnY%3DERNXZDAVTtHUpsfe4NoDg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Stop words returning results?

2015-04-02 Thread Rupert Smith

Hi,

I think I need to understand how the "_all" field works when it comes to
analysis. I want to query against all field that are indexed, not just
specific ones, this is for a generic free text search against content.

I have created a stopword filter with just "the" and "and" in it.

I can query how the fields are analyzed and find that "the" is indeed
removed:

curl http://localhost:9200/content/_analyze?field=content -d "the"
{"tokens":[]}

But if I check how the _all field is analyzed:

curl http://localhost:9200/content/_analyze?field=_all -d "the"
{"tokens":[{"token":"the","start_offset":0,"end_offset":3,"type":"","position":1}]}

So when I query _all I get results that I don't want.

Here is the start of my mappings, as you can see I did not set anything on
_all.

{"content":{"mappings":{"business_rates_calculator":{
"dynamic":"false",
"properties":{
"businessTypes":{"type":"string","index":"not_analyzed"},
"content":{"type":"string","analyzer":"custom_english"},
...

How do I set up _all so that it is composed of all the fields that are
mapped, each analyzed by the analyzer configured on the field. I assumed
this is how _all would work by default, but clearly not.

Thanks for your assistance.

Rupert

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0cee71aa-891b-4f11-9e3d-8510ac34a5fe%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

DeleteByID: best solution?

2015-04-02 Thread ku3ia

Hi all!
Please, advise.
I have a cluster, let say, with 5 indices. Each index contains two types. I
need to delete some records. But I only know the ID of these records. I
haven't any information about index and type to which these records belong.
So my question is: how to delete records in fast and light way via bulk api.

Thanks for any help.



--
View this message in context: 
http://elasticsearch-users.115913.n3.nabble.com/DeleteByID-best-solution-tp4073082.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1427964439307-4073082.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.

Re: Expand sharding on exist index

2015-04-02 Thread David Pilato

No.

But you can create a new index with more shards and if you are using an alias 
link it to the two indices.
That’s how you can add more shards to an index (virtual index aka alias)

Or you need to reindex all from the old index to the new index.


HTH

-- 
David Pilato - Developer | Evangelist 
elastic.co
@dadoonet  | @elasticsearchfr 
 | @scrutmydocs 






> Le 2 avr. 2015 à 10:31, Grigory Rubstein  a écrit :
> 
> Hi
> 
> Is its possible to change number of shards on exist index, where part of the 
> types does not stored fields ?
> 
> This message may contain confidential and/or privileged information. 
> If you are not the addressee or authorized to receive this on behalf of the 
> addressee you must not use, copy, disclose or take action based on this 
> message or any information herein. 
> If you have received this message in error, please advise the sender 
> immediately by reply email and delete this message. Thank you.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearch+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/af54f124-9480-407b-bf08-a9edbf8460d3%40googlegroups.com
>  
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/983AFEAC-9310-4BC6-AC9E-7E97E58C6D8A%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Expand sharding on exist index

Hi

Is its possible to change number of shards on exist index, where part of 
the types does not stored fields ?

-- 
This message may contain confidential and/or privileged information. 
If you are not the addressee or authorized to receive this on behalf of the 
addressee you must not use, copy, disclose or take action based on this 
message or any information herein. 
If you have received this message in error, please advise the sender 
immediately by reply email and delete this message. Thank you.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/af54f124-9480-407b-bf08-a9edbf8460d3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Multiple sub aggregation in java api