Re: Elasticsearch : Need advice on architectural design of my cluster

[Mandeep Gulati]

Thanks for the response Mark!

However, I am trying to understand how massive index can be a problem if 
everytime I know which type to query ? Any explanation or link to some 
documentation regarding this ?

On Tuesday, December 30, 2014 3:42:20 AM UTC+5:30, Mark Walkom wrote:
>
> Ideally you want to keep different types in different indexes.
> And you definitely don't want everything in one massive index as that 
> won't scale well.
>
> On 28 December 2014 at 22:41, Mandeep Gulati  > wrote:
>
>> I am quite new to elasticsearch. I need to build a search system using 
>> the data from MongoDB. So, here is a high level overview of my application:
>>
>>- There are different users belonging to different organizations
>>- A User can upload multiple datasets. Each dataset is stored as a 
>>single document in MongoDB. However, each dataset contains an array of 
>>nodes which contain the data we are interested in.
>>- User can load one dataset at a time to his workspace and view the 
>>entire data for that particular dataset. But at a time, one user can view 
>>only one dataset. So, datasets are independent from each other and we 
>> never 
>>need to have any aggregation on multiple datasets.
>>- User can perform a search in a dataset which is loaded in his/her 
>>workspace. Search should return the matching elements from the nodes 
>> array 
>>of the dataset
>>
>> For illustration, here is a single doc in MongoDB datasets collection
>>
>> {
>>   "_id": ObjectId()
>>   "setName": "dummy_set",
>>   "nodes": [
>> {
>>   "id": ObjectId(),
>>   "label": "some text",
>>   "content" : "more text"
>> },
>> . . . 
>>   ]
>> }
>>
>> For this, the design that I have though about is:
>>
>>- There will be one index in my cluster
>>- Each single dataset will be stored in a separate type in the index. 
>>Name of the type will be the ObjectId of the dataset in mongoDB
>>- Each element in the nodes array of dataset will become a single 
>>document in the corresponding type in elasticsearch.
>>- I will use custom routing to make sure a single dataset resides on 
>>one shard only. For that, I will be using the type name (ObjectId of 
>>dataset from MongoDB) as my routing key. I assume, I will have to store 
>> it 
>>with each document in elasticsearch?
>>
>> Now I need to know if I am heading in a right direction ? Does the 
>> solution look scalable or is there something terribly wrong with the design 
>> and would love to hear some suggestions on how to improve it.
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/a177625f-2462-40c4-aad2-514ee3553b64%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/13880f3a-fe01-4d81-99eb-5c05f709b866%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch RPM doesn't create ES_USER nor ES_GROUP defaults?

[Mark Walkom]

I just installed ES 1.4.2 from repos on CentOS and it created both user and
group;
[root@vagrant-centos65 ~]# getent passwd|grep elasticsearch
elasticsearch:x:497:497:elasticsearch
user:/usr/share/elasticsearch:/sbin/nologin

It also set the directories it needs to write to to the correct permissions;
[root@vagrant-centos65 ~]# ll -d /var/lib/elasticsearch
drwxr-xr-x 2 elasticsearch elasticsearch 4096 Dec 16 14:12
/var/lib/elasticsearch

The config files however are owned by root, which is standard.

On 31 December 2014 at 05:30,  wrote:

> I've just installed Elasticsearch 1.4.2 using the RHEL RPM.  I noticed
> that it didn't create a new user "elasticsearch" nor did it create a new
> group "elasticsearch".  Nor are any of the directories I'd expect the
> default user to be able to read and write owned by anything other than root.
>
>
> This doesn't seem correct, at least from my perspective.  Won't the
> service not run correctly without having the default user and group already
> existing?  Do I have to manually create the elasticsearch user and group?
> What directories should I have that user/group own?  Are they're other
> changes that I have to manually make?
>
> Thanks,
> S.J.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/ec051a19-d46a-4516-a9ae-5abfebb8b16f%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_-y%3Dh8bWz5-R%2BtZLyC7E7A-2KGWYHpbqKgx9WjyLxrrQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to compare data based on different dates Using Kibana3

[Ramakrishna N]

I am keeping document counts on Y-Axis and Hour of Day(0-23) on X-Axis.
Now in the graph, I need to show hourly data for Dec 10 2014 and Dec 15 
2014.

Appreciate your help!

Regards,
Rama

On Tuesday, December 30, 2014 2:46:54 PM UTC-8, Ramakrishna N wrote:
>
> I am started using Kibana4, still I couldn't find a way to get this report.
> Could you please pont me to some example, if any?
>
>
> On Tuesday, December 30, 2014 10:04:26 AM UTC-8, Elvar Böðvarsson wrote:
>>
>> This will be possible in Kibana4 with scripted fields
>>
>> On Monday, December 29, 2014 9:22:51 PM UTC, Ramakrishna N wrote:
>>>
>>> Hi,
>>>
>>> I have a question regarding the ElasticSearch and Kibana.
>>>
>>> My goal is I have to compare two different dates of data by hour/min/sec.
>>>
>>> All my data is indexed and everything in single type.
>>>
>>> Regards,
>>> Ramakrishna Namburu
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/df9a4bbd-afeb-411c-a72e-8f6a5ee18916%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Slow Commands with 1.2.4 to 1.4.2 Upgrade

[Mark Walkom]

How slow?
Is the load on your system high?

On 31 December 2014 at 05:04,  wrote:

> I have about 50 GB of data (1 mil docs) in a single node--8 cores with 32
> GB (24 GB heap). I just upgraded from 1.2.4 to 1.4.2, and I noticed that a
> few commands take a long time to return, and marvel doesn't work as well as
> it used to.
>
> Some of the commands that are slow for me are _cat/indices and _nodes.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/f9ab96bf-b5c3-4f99-9c9c-e00568aada9c%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_6dPyKpiX_vrTnvybOoYLEOcKG5jatVtjy4SVht87fMQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Different indexes for different Panels in Kibana

[Mark Walkom]

As you have found so far, you cannot do this.

On 30 December 2014 at 19:44, Ashutosh Parab  wrote:

> I want to configure Kibana in such a way that my different panels have
> different indexes. For example, histogram panel uses index 'X'' and table
> panel uses index 'Y'.
> Is there any way to do this in Kibana 3. As far as I  checked, we can set
> Index only at Dashboard level and not at Panel level.
> Please advice
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/1405a67e-9572-4a16-b4ce-730b160f9d97%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_%3DjkSwg42%3D8qapZd6gYF36GyrgPQWAm2-y8A7G4TV49Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to compare data based on different dates Using Kibana3

[Ramakrishna N]

I am started using Kibana4, still I couldn't find a way to get this report.
Could you please pont me to some example, if any?


On Tuesday, December 30, 2014 10:04:26 AM UTC-8, Elvar Böðvarsson wrote:
>
> This will be possible in Kibana4 with scripted fields
>
> On Monday, December 29, 2014 9:22:51 PM UTC, Ramakrishna N wrote:
>>
>> Hi,
>>
>> I have a question regarding the ElasticSearch and Kibana.
>>
>> My goal is I have to compare two different dates of data by hour/min/sec.
>>
>> All my data is indexed and everything in single type.
>>
>> Regards,
>> Ramakrishna Namburu
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/afcfa844-dd7c-4b31-b4b0-110f6c400ba0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Java -d64 Option

[Ingebrigt]

The -d64 parameter is not working, see:

https://github.com/elasticsearch/elasticsearch/issues/6528

cheers,

Ingebrigt



--
View this message in context: 
http://elasticsearch-users.115913.n3.nabble.com/Java-d64-Option-tp3206638p4068321.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1419973644689-4068321.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.

Re: using a nested object field in a multi_match query

[thomas . vaughan]

On Wednesday, December 10, 2014 4:33:12 PM UTC-3, thomas@beatport.com 
wrote:
>
>
>
> On Monday, August 11, 2014 1:29:56 PM UTC-4, Mike Topper wrote:
>>
>> Hello,
>>
>> I'm having trouble coming up with how to supply a field within a nested 
>> object in the multi_match fields list.  I'm using the multi_match query in 
>> order to perform query time field boosting, but something like:
>>
>>
>>   "query": {
>> "multi_match": {
>>   "query": "China Mieville",
>>   "operator": "and",
>>   "fields": [
>> "_all", "title^2", "author.name^1.5"
>>   ]
>> }
>>   }
>>
>> doesn't seem to work.  the title is boosted fine but in fact if i take 
>> out the "_all" field then i can see that author.name is never being 
>> used.  is there a way to supply nested fields within a multi_match query?
>>
>
> I've just been bit by this too. Anyone know how to make this work?
>

In our case we switched the mapping type from "nested" to "object" and then 
this worked. I'm aware of the implications of this switch. We don't need 
the features provided by "nested". Others may, of course.

Thanks.

-Tom

 

>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/857a9674-4661-4730-9ec8-79ba3426a603%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: how to enter multi valued geo_point attribute in elastic search

[Nicholas Knize]

Hello Abhimanyu,  

Below is a simple example for creating a geo_shape mapping and inserting a 
MultiPoint document.  For further detail on inserting different geometry 
types be sure to have a look at 
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-geo-shape-type.html
 

# create index
curl -XPUT "http://localhost:9200/geoshapes?pretty=true"; -d'
{ 
  "mappings": {
"venues": {
  "properties": {
"name" : {
  "type" : "string"
}, 
"location" : {
  "type" : "geo_shape",
  "tree" : "quadtree",
  "precision" : "1m"
}
  }
}
  }
}';

# insert a multi point
curl -XPUT "http://localhost:9200/geoshapes/venues/1?pretty=true' -d '
{
  "name": "Chipotle Mexican Grill", 
  "location": { 
"type" : "MultiPoint", 
"coordinates": [
  [-74.011, 40.715],
  [-74.123, 41.325],
  [-73.224, 41.522]
]
  }
}'

# query all
curl -XGET "http://localhost:9200/geoshapes/venues/_search?pretty=true"; -d '
{
  "query": {
"match_all":{} 
  }
}'




On Tuesday, December 30, 2014 1:17:10 AM UTC-6, Abhimanyu Nagrath wrote:
>
> Hi, 
> I want to know the way to enter multi valued geo_point attribute in 
> elastic search?at present i am using 
> [{"lat":-40.2345,"lon":-30.2345},{"lat":-25.5678,"lon":-23.6789}] i am able 
> to enter but i am not able see the data entry on elastic search UI and also 
> not able to query.So please tell me that wether I am doing any thig wrong 
> or is there any other way.
> Thank You
> Abhimanyu
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/75d8c2a6-8f46-4df1-8383-53996722387d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Elasticsearch RPM doesn't create ES_USER nor ES_GROUP defaults?

[scottj1]

I've just installed Elasticsearch 1.4.2 using the RHEL RPM.  I noticed that 
it didn't create a new user "elasticsearch" nor did it create a new group 
"elasticsearch".  Nor are any of the directories I'd expect the default 
user to be able to read and write owned by anything other than root.


This doesn't seem correct, at least from my perspective.  Won't the service 
not run correctly without having the default user and group already 
existing?  Do I have to manually create the elasticsearch user and group?  
What directories should I have that user/group own?  Are they're other 
changes that I have to manually make?

Thanks,
S.J.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/ec051a19-d46a-4516-a9ae-5abfebb8b16f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Slow Commands with 1.2.4 to 1.4.2 Upgrade

[pskieu]

I have about 50 GB of data (1 mil docs) in a single node--8 cores with 32 
GB (24 GB heap). I just upgraded from 1.2.4 to 1.4.2, and I noticed that a 
few commands take a long time to return, and marvel doesn't work as well as 
it used to.

Some of the commands that are slow for me are _cat/indices and _nodes.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f9ab96bf-b5c3-4f99-9c9c-e00568aada9c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to compare data based on different dates Using Kibana3

[Elvar Böðvarsson]

This will be possible in Kibana4 with scripted fields

On Monday, December 29, 2014 9:22:51 PM UTC, Ramakrishna N wrote:
>
> Hi,
>
> I have a question regarding the ElasticSearch and Kibana.
>
> My goal is I have to compare two different dates of data by hour/min/sec.
>
> All my data is indexed and everything in single type.
>
> Regards,
> Ramakrishna Namburu
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/fde48619-c6bf-45ab-94db-819c170e3c35%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: "Aggregations" without doc-counts

[Adrien Grand]

Hi Eliott,

The overhead of computing the doc counts is actually low, I don't think you
should worry about it.

On Tue, Dec 30, 2014 at 5:12 PM, Elliott Bradshaw 
wrote:

> Hi,
>
> I'm currently working on a project that visualizes geospatial data in
> Elasticsearch.  One of the things I am doing is generating heatmaps with
> the geohash grid aggregation.  I would like to take this to the extreme
> case of gridding down to the individual pixel level to display raster
> images of a data set, but I am not concerned with the total doc count of
> each geohash.  Is there a way (or could it be implemented) where an
> optimized aggregation could be run that simply lists the existing terms
> (geohashes) and does not bother with aggregating their counts?  If this
> significantly improved performance, such a feature would be very valuable.
>
> Thanks!
>
> - Elliott Bradshaw
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/834ebcb1-43b3-486d-bd1a-952005a6a66d%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Adrien Grand

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j45%2BgfqBk73Mfh_b6JVLcG9E7RfkE9eovPgL5kYG%3DzRug%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Arrays and Kibana

[Guillermo Ortiz]

Is there any way to see the fields of an array in a pretty way in Kibana or 
it isn't supported?? Now, I just see them like just one field.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5e83509a-2928-4659-b6e8-f00e870e4f59%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

"Aggregations" without doc-counts

[Elliott Bradshaw]

Hi,

I'm currently working on a project that visualizes geospatial data in 
Elasticsearch.  One of the things I am doing is generating heatmaps with 
the geohash grid aggregation.  I would like to take this to the extreme 
case of gridding down to the individual pixel level to display raster 
images of a data set, but I am not concerned with the total doc count of 
each geohash.  Is there a way (or could it be implemented) where an 
optimized aggregation could be run that simply lists the existing terms 
(geohashes) and does not bother with aggregating their counts?  If this 
significantly improved performance, such a feature would be very valuable.

Thanks!

- Elliott Bradshaw

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/834ebcb1-43b3-486d-bd1a-952005a6a66d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elastic and Kibana, indexing al Json with an field array looks like a plain String.

[Guillermo Ortiz]

I was reading documentation about Kibana and ES and I think that Elastic 
doesn't type the nested properties to indicate that some of my fields are 
nested. Anyway, I created another index to try it with nested properties 
and I can't see them on the field panel from Kibana.

On Tuesday, December 30, 2014 1:32:14 PM UTC+1, Guillermo Ortiz wrote:
>
> I'm starting working with Elastic and Kibana.
> I have indexed some JSON but I think that there're something that I'm 
> missing.
>
> My JSONs have some simple fields and another field that it's an array, the 
> schema is:
>
> {"akamai":{"mappings":{"miIndex":{"properties":{"date":{"type":"long"},"logs":{"properties":{"message":{"properties":{"cliIP":{"type":"string"},"fwdHost":{"type":"string"},"reqHost":{"type":"string"}}},"waf":{"properties":{"warnSlrs":{"type":"string"},"type":{"type":"string"}}
>
>
> Fields as type, date are independent, but "logs" it's just a string in Kibana.
>
> I'd like to see it like a real JSON as tree or something similar. I'm 
> confused because the schema looks good, but in Kibana it's just a String. 
>
> I have some Java classes to model the Json and use Jackson to parse it as 
> this:
>
>
> IndexResponse response = client.prepareIndex(indexName, docType)
>   .setSource(jsonToIndex).execute().actionGet();
>
>
> I thought as well that it could be some chars what are not correctly escaped 
> so I used StringEscapeUtils to escape them, but it doesn't fix anything. 
>
> Should I be able to see the "logs" field as a json or arrays in Kibana can't 
> be explored as a real json? what am I missing?
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/161f1e10-ccb7-4207-95e7-ff2f07dc70cc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ES upgrade 0.20.6 to 1.3.4 -> CorruptIndexException

[Robert Muir]

Yes. again, use the latest version (1.4.x). its very simple.

On Tue, Dec 30, 2014 at 8:53 AM, Georgeta Boanea  wrote:
> The Lucene bug is referring to 3.0-3.3 versions, Elasticsearch 0.20.6 is
> using Lucene 3.6, is it the same bug?
>
>
> On Tuesday, December 30, 2014 2:08:48 PM UTC+1, Robert Muir wrote:
>>
>> This bug occurs because you are upgrading to an old version of
>> elasticsearch (1.3.4). Try the latest version where the bug is fixed:
>> https://issues.apache.org/jira/browse/LUCENE-5975
>>
>> On Fri, Dec 19, 2014 at 5:40 AM, Georgeta Boanea  wrote:
>> > Hi All,
>> >
>> > After upgrading from ES 0.20.6 to 1.3.4 the following messages occurred:
>> >
>> > [2014-12-19 10:02:06.714 GMT] WARN ||
>> > elasticsearch[es-node-name][generic][T#14]
>> > org.elasticsearch.cluster.action.shard  [es-node-name] [index-name][3]
>> > sending failed shard for [index-name][3], node[qOTLmb3IQC2COXZh1n9O2w],
>> > [P],
>> > s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard,
>> > message
>> > [IndexShardGatewayRecoveryException[[index-name][3] failed to fetch
>> > index
>> > version after copying it over]; nested:
>> > CorruptIndexException[[index-name][3] Corrupted index
>> > [corrupted_Ackui00SSBi8YXACZGNDkg] caused by: CorruptIndexException[did
>> > not
>> > read all bytes from file: read 112 vs size 113 (resource:
>> >
>> > BufferedChecksumIndexInput(NIOFSIndexInput(path="path/3/index/_uzm_2.del")))]];
>> > ]]
>> >
>> > [2014-12-19 10:02:08.390 GMT] WARN ||
>> > elasticsearch[es-node-name][generic][T#20]
>> > org.elasticsearch.indices.cluster
>> > [es-node-name] [index-name][3] failed to start shard
>> > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
>> > [index-name][3] failed to fetch index version after copying it over
>> > at
>> >
>> > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:152)
>> > at
>> >
>> > org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> > at java.lang.Thread.run(Thread.java:745)
>> > Caused by: org.apache.lucene.index.CorruptIndexException:
>> > [index-name][3]
>> > Corrupted index [corrupted_Ackui00SSBi8YXACZGNDkg] caused by:
>> > CorruptIndexException[did not read all bytes from file: read 112 vs size
>> > 113
>> > (resource:
>> >
>> > BufferedChecksumIndexInput(NIOFSIndexInput(path="path/3/index/_uzm_2.del")))]
>> > at org.elasticsearch.index.store.Store.failIfCorrupted(Store.java:353)
>> > at org.elasticsearch.index.store.Store.failIfCorrupted(Store.java:338)
>> > at
>> >
>> > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:119)
>> > ... 4 more
>> >
>> > Shard [3] of the index remains unallocated and the cluster remains in a
>> > RED
>> > state.
>> >
>> > curl -XGET 'http://localhost:48012/_cluster/health?pretty=true'
>> > {
>> >   "cluster_name" : "cluster-name",
>> >   "status" : "red",
>> >   "timed_out" : false,
>> >   "number_of_nodes" : 5,
>> >   "number_of_data_nodes" : 5,
>> >   "active_primary_shards" : 10,
>> >   "active_shards" : 20,
>> >   "relocating_shards" : 0,
>> >   "initializing_shards" : 1,
>> >   "unassigned_shards" : 1
>> > }
>> >
>> > If I do an optimize (curl -XPOST
>> > http://localhost:48012/index-name/_optimize?max_num_segments=1) for the
>> > index before the update, everything is fine. Optimize works just before
>> > the
>> > update, if is done after the update the problem remains the same.
>> >
>> > Any idea why this problem occurs?
>> > Is there another way to avoid this problem? I want to avoid optimize in
>> > case
>> > of large volume of data.
>> >
>> > Thank you,
>> > Georgeta
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "elasticsearch" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > email to elasticsearc...@googlegroups.com.
>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/elasticsearch/74d0af86-c661-4e58-ba2c-d38adde1291c%40googlegroups.com.
>> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/39216d8f-da8e-4793-abcc-dd004586d45f%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 

disable cluster rebalancing for replicas

[Arun]

Hello,

   I have a 3-node ES (v1.2.1) cluster with 1 replica. When there is a node 
failure, the cluster rebalances and allocates primary shards(6 per index) 
and replicas among the remaining 2 nodes which is fine. But we had an issue 
where the disk space filled up and brought the whole cluster down. I 
understand that we should allocate enough disk space to all nodes, but is 
there a way to disable re-allocation of replicas ? On a node failure, we 
want the replica to get promoted to primary, but we don't want new replicas 
to be allocated among the remaining 2 nodes. Replicas should get allocated 
when the cluster has 3 members. I have looked at 
cluster.routing.allocation*rebalance 
options but I'm confused. Any help is appreciated.

Thanks,
Arun

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/ca609103-1921-4702-97e0-62aaf023803b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ES upgrade 0.20.6 to 1.3.4 -> CorruptIndexException

[Georgeta Boanea]

The Lucene bug is referring to 3.0-3.3 versions, Elasticsearch 0.20.6 is 
using Lucene 3.6, is it the same bug?


On Tuesday, December 30, 2014 2:08:48 PM UTC+1, Robert Muir wrote:
>
> This bug occurs because you are upgrading to an old version of 
> elasticsearch (1.3.4). Try the latest version where the bug is fixed: 
> https://issues.apache.org/jira/browse/LUCENE-5975 
>
> On Fri, Dec 19, 2014 at 5:40 AM, Georgeta Boanea  > wrote: 
> > Hi All, 
> > 
> > After upgrading from ES 0.20.6 to 1.3.4 the following messages occurred: 
> > 
> > [2014-12-19 10:02:06.714 GMT] WARN || 
> > elasticsearch[es-node-name][generic][T#14] 
> > org.elasticsearch.cluster.action.shard  [es-node-name] [index-name][3] 
> > sending failed shard for [index-name][3], node[qOTLmb3IQC2COXZh1n9O2w], 
> [P], 
> > s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, 
> message 
> > [IndexShardGatewayRecoveryException[[index-name][3] failed to fetch 
> index 
> > version after copying it over]; nested: 
> > CorruptIndexException[[index-name][3] Corrupted index 
> > [corrupted_Ackui00SSBi8YXACZGNDkg] caused by: CorruptIndexException[did 
> not 
> > read all bytes from file: read 112 vs size 113 (resource: 
> > 
> BufferedChecksumIndexInput(NIOFSIndexInput(path="path/3/index/_uzm_2.del")))]];
>  
>
> > ]] 
> > 
> > [2014-12-19 10:02:08.390 GMT] WARN || 
> > elasticsearch[es-node-name][generic][T#20] 
> org.elasticsearch.indices.cluster 
> > [es-node-name] [index-name][3] failed to start shard 
> > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: 
> > [index-name][3] failed to fetch index version after copying it over 
> > at 
> > 
> org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:152)
>  
>
> > at 
> > 
> org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
>  
>
> > at 
> > 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  
>
> > at 
> > 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  
>
> > at java.lang.Thread.run(Thread.java:745) 
> > Caused by: org.apache.lucene.index.CorruptIndexException: 
> [index-name][3] 
> > Corrupted index [corrupted_Ackui00SSBi8YXACZGNDkg] caused by: 
> > CorruptIndexException[did not read all bytes from file: read 112 vs size 
> 113 
> > (resource: 
> > 
> BufferedChecksumIndexInput(NIOFSIndexInput(path="path/3/index/_uzm_2.del")))] 
>
> > at org.elasticsearch.index.store.Store.failIfCorrupted(Store.java:353) 
> > at org.elasticsearch.index.store.Store.failIfCorrupted(Store.java:338) 
> > at 
> > 
> org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:119)
>  
>
> > ... 4 more 
> > 
> > Shard [3] of the index remains unallocated and the cluster remains in a 
> RED 
> > state. 
> > 
> > curl -XGET 'http://localhost:48012/_cluster/health?pretty=true' 
> > { 
> >   "cluster_name" : "cluster-name", 
> >   "status" : "red", 
> >   "timed_out" : false, 
> >   "number_of_nodes" : 5, 
> >   "number_of_data_nodes" : 5, 
> >   "active_primary_shards" : 10, 
> >   "active_shards" : 20, 
> >   "relocating_shards" : 0, 
> >   "initializing_shards" : 1, 
> >   "unassigned_shards" : 1 
> > } 
> > 
> > If I do an optimize (curl -XPOST 
> > http://localhost:48012/index-name/_optimize?max_num_segments=1) for the 
> > index before the update, everything is fine. Optimize works just before 
> the 
> > update, if is done after the update the problem remains the same. 
> > 
> > Any idea why this problem occurs? 
> > Is there another way to avoid this problem? I want to avoid optimize in 
> case 
> > of large volume of data. 
> > 
> > Thank you, 
> > Georgeta 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups 
> > "elasticsearch" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an 
> > email to elasticsearc...@googlegroups.com . 
> > To view this discussion on the web visit 
> > 
> https://groups.google.com/d/msgid/elasticsearch/74d0af86-c661-4e58-ba2c-d38adde1291c%40googlegroups.com.
>  
>
> > For more options, visit https://groups.google.com/d/optout. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/39216d8f-da8e-4793-abcc-dd004586d45f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [IMPORTANT] Issues using Perl API client installation

[Clinton Gormley]

On Monday, 29 December 2014 22:26:24 UTC+1, Vilas Reddy wrote:
>
> Hi,
>
> I am trying to use Perl API for retrieving data from Elasticsearch.
> I am using Elasticsearch in windows cygwin.
>
> I need help with installing perl api and using it. I tried the following:
>
> *1. Installed cpan in cygwin and tried installing using cpan 
> Search::Elasticsearch. I get the following error*:
>

You may find it easier to use Strawberry Perl for windows 
http://strawberryperl.com/ instead of cygwin



>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/54feaf17-ed01-4147-8d68-fed11e35e6cf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [IMPORTANT] Issues using Perl API client installation

[joergpra...@gmail.com]

This is a Cygwin setup problem. Although off-topic, I recommend to run

cygcheck -s -v -r -h > cygcheck_output.txt

and post the result text somewhere so Cygwin experts can examine it.

See also https://cygwin.com/cygwin-ug-net/using-utils.html

Jörg

On Mon, Dec 29, 2014 at 11:25 PM, Vilas Reddy  wrote:

> Thanks Jörg for your prompt reply.
>
> Can you please elaborate on how to do this. As i mentioned i am using
> cygwin.
>
> Regards,
> Vilas
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/a8d16219-beb9-4cdc-9abc-422efbfe7b3a%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE3qAL_WAfroTRcp1j6zBQAbuFQJGz18hEsPWj-3umdEQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: ES upgrade 0.20.6 to 1.3.4 -> CorruptIndexException

[Robert Muir]

This bug occurs because you are upgrading to an old version of
elasticsearch (1.3.4). Try the latest version where the bug is fixed:
https://issues.apache.org/jira/browse/LUCENE-5975

On Fri, Dec 19, 2014 at 5:40 AM, Georgeta Boanea  wrote:
> Hi All,
>
> After upgrading from ES 0.20.6 to 1.3.4 the following messages occurred:
>
> [2014-12-19 10:02:06.714 GMT] WARN ||
> elasticsearch[es-node-name][generic][T#14]
> org.elasticsearch.cluster.action.shard  [es-node-name] [index-name][3]
> sending failed shard for [index-name][3], node[qOTLmb3IQC2COXZh1n9O2w], [P],
> s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message
> [IndexShardGatewayRecoveryException[[index-name][3] failed to fetch index
> version after copying it over]; nested:
> CorruptIndexException[[index-name][3] Corrupted index
> [corrupted_Ackui00SSBi8YXACZGNDkg] caused by: CorruptIndexException[did not
> read all bytes from file: read 112 vs size 113 (resource:
> BufferedChecksumIndexInput(NIOFSIndexInput(path="path/3/index/_uzm_2.del")))]];
> ]]
>
> [2014-12-19 10:02:08.390 GMT] WARN ||
> elasticsearch[es-node-name][generic][T#20] org.elasticsearch.indices.cluster
> [es-node-name] [index-name][3] failed to start shard
> org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
> [index-name][3] failed to fetch index version after copying it over
> at
> org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:152)
> at
> org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.lucene.index.CorruptIndexException: [index-name][3]
> Corrupted index [corrupted_Ackui00SSBi8YXACZGNDkg] caused by:
> CorruptIndexException[did not read all bytes from file: read 112 vs size 113
> (resource:
> BufferedChecksumIndexInput(NIOFSIndexInput(path="path/3/index/_uzm_2.del")))]
> at org.elasticsearch.index.store.Store.failIfCorrupted(Store.java:353)
> at org.elasticsearch.index.store.Store.failIfCorrupted(Store.java:338)
> at
> org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:119)
> ... 4 more
>
> Shard [3] of the index remains unallocated and the cluster remains in a RED
> state.
>
> curl -XGET 'http://localhost:48012/_cluster/health?pretty=true'
> {
>   "cluster_name" : "cluster-name",
>   "status" : "red",
>   "timed_out" : false,
>   "number_of_nodes" : 5,
>   "number_of_data_nodes" : 5,
>   "active_primary_shards" : 10,
>   "active_shards" : 20,
>   "relocating_shards" : 0,
>   "initializing_shards" : 1,
>   "unassigned_shards" : 1
> }
>
> If I do an optimize (curl -XPOST
> http://localhost:48012/index-name/_optimize?max_num_segments=1) for the
> index before the update, everything is fine. Optimize works just before the
> update, if is done after the update the problem remains the same.
>
> Any idea why this problem occurs?
> Is there another way to avoid this problem? I want to avoid optimize in case
> of large volume of data.
>
> Thank you,
> Georgeta
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/74d0af86-c661-4e58-ba2c-d38adde1291c%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAMUKNZW3Kc-8smWQjn1VRrk2yhgdiA33EctWUiXEOkxg46BjiQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Elastic and Kibana, indexing al Json with an field array looks like a plain String.

[Guillermo Ortiz]

I'm starting working with Elastic and Kibana.
I have indexed some JSON but I think that there're something that I'm 
missing.

My JSONs have some simple fields and another field that it's an array, the 
schema is:

{"akamai":{"mappings":{"miIndex":{"properties":{"date":{"type":"long"},"logs":{"properties":{"message":{"properties":{"cliIP":{"type":"string"},"fwdHost":{"type":"string"},"reqHost":{"type":"string"}}},"waf":{"properties":{"warnSlrs":{"type":"string"},"type":{"type":"string"}}


Fields as type, date are independent, but "logs" it's just a string in Kibana.

I'd like to see it like a real JSON as tree or something similar. I'm confused 
because the schema looks good, but in Kibana it's just a String. 

I have some Java classes to model the Json and use Jackson to parse it as this:


IndexResponse response = client.prepareIndex(indexName, docType)
.setSource(jsonToIndex).execute().actionGet();


I thought as well that it could be some chars what are not correctly escaped so 
I used StringEscapeUtils to escape them, but it doesn't fix anything. 

Should I be able to see the "logs" field as a json or arrays in Kibana can't be 
explored as a real json? what am I missing?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f21d4be2-0e13-4f39-926c-690e0f65843a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: elasticsearch java.lang.ArrayIndexOutOfBoundsException: 1

[David Pilato]

I would first try without any script.
If it still does not work, you should may be open an issue in the project.

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

> Le 30 déc. 2014 à 11:00, Vinay H M  a écrit :
> 
> ya shell script can be used ... but this error is occurring when i run the 
> elasticsearch... can i get the one sample code to extract the csv file and
> to execute in elasticsearch and kibana 
> 
>> On Tuesday, December 30, 2014 1:25:03 PM UTC+5:30, David Pilato wrote:
>> I don't know this plugin but are you sure you can provide a shell script?
>> Sounds like Groovy is trying to execute it...
>> 
>> --
>> David ;-)
>> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>> 
>>> Le 30 déc. 2014 à 04:57, Vinay H M  a écrit :
>>> 
>>> 
>>> 
 On Tuesday, December 30, 2014 9:23:58 AM UTC+5:30, Vinay H M wrote:
 Hi All
 
 I found the error while running the elasticsearch ..plzz some one solve it 
 
 
 [2014-12-30 
 09:16:22,389][ERROR][org.agileworks.elasticsearch.river.csv.CSVRiver] 
 [Aliyah Bishop] [csv][my_csv_river] Error has occured during processing 
 file 'PDUserDeviceDataTable.csv.processing' , skipping line: 
 '[249573";"875";"testaasim";"00:12:F3:1B:A5:68";"2";"1344";"0";"29.7";"58.3";"1419835852";"20.0";"30.0";"40.0";"50.0";"500";"500";"12.9226205";"77.5605173]'
  and continue in processing
 java.lang.ArrayIndexOutOfBoundsException: 1
at 
 org.codehaus.groovy.runtime.BytecodeInterface8.objectArrayGet(BytecodeInterface8.java:360)
at 
 org.agileworks.elasticsearch.river.csv.OpenCSVFileProcessor.processDataLine(OpenCSVFileProcessor.groovy:72)
at 
 org.agileworks.elasticsearch.river.csv.OpenCSVFileProcessor.this$2$processDataLine(OpenCSVFileProcessor.groovy)
at 
 org.agileworks.elasticsearch.river.csv.OpenCSVFileProcessor$this$2$processDataLine.callCurrent(Unknown
  Source)
at 
 org.agileworks.elasticsearch.river.csv.OpenCSVFileProcessor.process(OpenCSVFileProcessor.groovy:49)
at 
 org.agileworks.elasticsearch.river.csv.CSVConnector.processAllFiles(CSVConnector.groovy:47)
at 
 org.agileworks.elasticsearch.river.csv.CSVConnector.run(CSVConnector.groovy:20)
at java.lang.Thread.run(Thread.java:745)
>>> 
>>> 
>>> the command i am using to create index 
>>> 
>>> curl -XPUT localhost:9200/_river/my_csv_river/_meta -d '
>>> {
>>> "type" : "csv",
>>> "csv_file" : {
>>> "folder" : "/home/paqs/Downloads/kibana/dec",
>>> "filename_pattern" : ".*\\.csv$",
>>> "poll":"1m",
>>> "fields" : [
>>>"Sno",
>>>"userld",
>>>"userName",
>>>"deviceld",
>>>"deviceCurrentMode",
>>>"co2Level",
>>>"dustLevel",
>>>"temperature",
>>>"relativeHumidity",
>>>"timeStamp",
>>>"tempLow",
>>>"tempHigh",
>>>"rhLow",
>>>"rhHigh",
>>>"dust",
>>>"pollution",
>>>"latitude",
>>>"longitude"
>>> ],
>>> "first_line_is_header" : "false",
>>> "field_separator" : ",",
>>> "escape_character" : "\\",
>>> "quote_character" : "\"",
>>> "field_id" : "id",
>>> "field_timestamp" : "imported_at",
>>> "concurrent_requests" : "1",
>>> "charset" : "UTF-8",
>>> "script_before_file": 
>>> "/home/paqs/Downloads/kibana/dec/before_file.sh",
>>> "script_after_file": 
>>> "/home/paqs/Downloads/kibana/dec/after_file.sh",
>>> "script_before_all": 
>>> "/home/paqs/Downloads/kibana/dec/before_all.sh",
>>> "script_after_all": "/home/paqs/Downloads/kibana/dec/after_all.sh"
>>> },
>>> "index" : {
>>> "index" : "decdevicedata",
>>> "type" : "alert",
>>> "bulk_size" : 1000,
>>> "bulk_threshold" : 10
>>> }
>>> }'
>>> 
>>> 
>>> the curl command i am using to create the mapping 
>>> 
>>> Create a mapping
>>> #
>>> curl -XPUT http://localhost:9200/decdevicedata -d '
>>> {
>>> "settings" : {
>>> "number_of_shards" : 1
>>> },
>>> "mappings" : {
>>> "alert" : {
>>> "properties" : {
>>> "Sno": {"type" : "integer"},
>>> "co2Level" : {"type" : "integer"},
>>> "deviceCurrentMode" : {"type" : "integer"},
>>> "deviceld"  : {"type" : "string"},
>>> "dust"  : {"type" : "integer"},
>>> "dustLevel" : {"type" : "integer"},
>>> "latitude": {"type" : "integer"},
>>> "longitude": {"type" : "integer"},
>>> "pollution" : {"type" : "integer"},
>>> "relativeHumidity" : {"type" : "float"},
>>> "rhLow": {"type" : "float"},
>>> "rhHigh": {"type" : "float"},
>>> "temperature": {"type" : "float"},
>>> "tempLow": {"type" :

Re: Visualize stats of MS SQL tables using ElasticSearch

[Elvar Böðvarsson]

1. JDBC River to get data into Elasticsearch
2. Kibana to visualize everything

On Tuesday, December 30, 2014 4:45:31 AM UTC, Ashutosh Parab wrote:
>
> What I am doing is loading my MS SQL database into ElasticSearch. I want 
> to perform different types of aggregations/statistics correaltions on the 
> rows of those tables. So I wanted to know whether there is any tool to 
> visualize such data. 
> Is there a tutorial to demonstrate how this can be done?
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/9fc4dfe2-daa8-4fed-bfe5-4bdaea4c9d6b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: elasticsearch java.lang.ArrayIndexOutOfBoundsException: 1

[Vinay H M]

ya shell script can be used ... but this error is occurring when i run the 
elasticsearch... can i get the one sample code to extract the csv file and 
   to execute in elasticsearch and kibana 

On Tuesday, December 30, 2014 1:25:03 PM UTC+5:30, David Pilato wrote:
>
> I don't know this plugin but are you sure you can provide a shell script?
> Sounds like Groovy is trying to execute it...
>
> --
> David ;-)
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>
> Le 30 déc. 2014 à 04:57, Vinay H M > a 
> écrit :
>
>
>
> On Tuesday, December 30, 2014 9:23:58 AM UTC+5:30, Vinay H M wrote:
>>
>> Hi All
>>
>> I found the error while running the elasticsearch ..plzz some one solve 
>> it 
>>
>>
>> [2014-12-30 
>> 09:16:22,389][ERROR][org.agileworks.elasticsearch.river.csv.CSVRiver] 
>> [Aliyah Bishop] [csv][my_csv_river] Error has occured during processing 
>> file 'PDUserDeviceDataTable.csv.processing' , skipping line: 
>> '[249573";"875";"testaasim";"00:12:F3:1B:A5:68";"2";"1344";"0";"29.7";"58.3";"1419835852";"20.0";"30.0";"40.0";"50.0";"500";"500";"12.9226205";"77.5605173]'
>>  
>> and continue in processing
>> java.lang.ArrayIndexOutOfBoundsException: 1
>> at 
>> org.codehaus.groovy.runtime.BytecodeInterface8.objectArrayGet(BytecodeInterface8.java:360)
>> at 
>> org.agileworks.elasticsearch.river.csv.OpenCSVFileProcessor.processDataLine(OpenCSVFileProcessor.groovy:72)
>> at 
>> org.agileworks.elasticsearch.river.csv.OpenCSVFileProcessor.this$2$processDataLine(OpenCSVFileProcessor.groovy)
>> at 
>> org.agileworks.elasticsearch.river.csv.OpenCSVFileProcessor$this$2$processDataLine.callCurrent(Unknown
>>  
>> Source)
>> at 
>> org.agileworks.elasticsearch.river.csv.OpenCSVFileProcessor.process(OpenCSVFileProcessor.groovy:49)
>> at 
>> org.agileworks.elasticsearch.river.csv.CSVConnector.processAllFiles(CSVConnector.groovy:47)
>> at 
>> org.agileworks.elasticsearch.river.csv.CSVConnector.run(CSVConnector.groovy:20)
>> at java.lang.Thread.run(Thread.java:745)
>>
>
>
> the command i am using to create index 
>
> curl -XPUT localhost:9200/_river/my_csv_river/_meta -d '
> {
> "type" : "csv",
> "csv_file" : {
> "folder" : "/home/paqs/Downloads/kibana/dec",
> "filename_pattern" : ".*\\.csv$",
> "poll":"1m",
> "fields" : [
>"Sno",
>"userld",
>"userName",
>"deviceld",
>"deviceCurrentMode",
>"co2Level",
>"dustLevel",
>"temperature",
>"relativeHumidity",
>"timeStamp",
>"tempLow",
>"tempHigh",
>"rhLow",
>"rhHigh",
>"dust",
>"pollution",
>"latitude",
>"longitude"
> ],
> "first_line_is_header" : "false",
> "field_separator" : ",",
> "escape_character" : "\\",
> "quote_character" : "\"",
> "field_id" : "id",
> "field_timestamp" : "imported_at",
> "concurrent_requests" : "1",
> "charset" : "UTF-8",
> "script_before_file": 
> "/home/paqs/Downloads/kibana/dec/before_file.sh",
> "script_after_file": 
> "/home/paqs/Downloads/kibana/dec/after_file.sh",
> "script_before_all": 
> "/home/paqs/Downloads/kibana/dec/before_all.sh",
> "script_after_all": "/home/paqs/Downloads/kibana/dec/after_all.sh"
> },
> "index" : {
> "index" : "decdevicedata",
> "type" : "alert",
> "bulk_size" : 1000,
> "bulk_threshold" : 10
> }
> }'
>
>
> the curl command i am using to create the mapping 
>
> Create a mapping
> #
> curl -XPUT http://localhost:9200/decdevicedata -d '
> {
> "settings" : {
> "number_of_shards" : 1
> },
> "mappings" : {
> "alert" : {
> "properties" : {
> "Sno": {"type" : "integer"},
> "co2Level" : {"type" : "integer"},
> "deviceCurrentMode" : {"type" : "integer"},
> "deviceld"  : {"type" : "string"},
> "dust"  : {"type" : "integer"},
> "dustLevel" : {"type" : "integer"},
> "latitude": {"type" : "integer"},
> "longitude": {"type" : "integer"},
> "pollution" : {"type" : "integer"},
> "relativeHumidity" : {"type" : "float"},
> "rhLow": {"type" : "float"},
> "rhHigh": {"type" : "float"},
> "temperature": {"type" : "float"},
> "tempLow": {"type" : "float"},
> "tempHigh": {"type" : "float"},
> "timeStamp" : {"type" : "date", "ignore_malformed" : true, 
> "format" : "dateOptionalTime"},
> "userld" : {"type" : "integer"},
> "userName" : {"type" : "string", "index" : "not_analyzed"}
> 
> }
> }
> }
> }'
>
>
>
>  
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from t

Re: copy_to and boost factor

[Preeti Jain]

Thanks Jörg.

On Tuesday, December 30, 2014 2:43:20 PM UTC+5:30, Jörg Prante wrote:

> Boosting is not supported in copy_to, see
>
> https://github.com/elasticsearch/elasticsearch/pull/4972
>
> Alternative, as quoted from the commit message
>
> "Instead, users should either (1) query the common field AND the field 
> that requires boosting, or (2) the multi_match query will soon be able to 
> perform term-centric cross-field matching that will allow per-field 
> boosting at query time (coming in 1.1)."
>
> See my multi_match gist for an example
>
> https://gist.github.com/jprante/85126685973311603c8a
>
> Jörg
>
> On Tue, Dec 30, 2014 at 9:36 AM, Preeti Jain  > wrote:
>
>> Hi,
>>  
>> Does the individual boost factor get honoured for fields getting copied 
>> into a custom _all field via copy_to option?
>> I am facing issues with the behavior. Have a look at the example below
>>  
>> PUT localhost:9200/investigations/TestCustom/_mapping
>> {
>>   "TestCustom": {
>>   "properties": { 
>>   "custom_all":{
>>   "type": "string"
>>   },
>>   "name":{
>>   "type": "string",
>>   "copy_to":"custom_all",
>>   "boost":2.0
>>   },
>> "surName":{
>>   "type": "string",
>>   "copy_to":"custom_all"
>>   },
>>   "dateRecorded":{
>>   "type": "date",
>>   "copy_to":"custom_all"
>>   },
>> "Id":{
>>   "type": "long",
>>   "copy_to":"custom_all"
>>   }
>>   }
>> }
>> }
>>  
>> Fiels "name" has been given a boost factor of 2.0
>>  
>> I post 2 documents now
>>  
>> POST localhost:9200/investigations/TestCustom/5
>> {
>> "surName":"Preeti"
>> }
>> POST localhost:9200/investigations/TestCustom/6
>> {
>> "name":"Preeti"
>> }
>>  
>> and then search the custom_all field
>>  
>> GET localhost:9200/investigations/TestCustom/_search
>> {
>> "query":{
>>"query_string": {
>>"default_field": "custom_all",
>>"query": "Preeti"
>> }
>> }
>> }
>>  
>> I expected document with id 6 to be the top result but the results were
>>  
>> {
>>"took": 40,
>>"timed_out": false,
>>"_shards": {
>>   "total": 5,
>>   "successful": 5,
>>   "failed": 0
>>},
>>"hits": {
>>   "total": 2,
>>   "max_score": 1,
>>   "hits": [
>>  {
>> "_index": "investigations",
>> "_type": "TestCustom",
>> "_id": "5",
>> "_score": 1,
>> "_source": {
>>"surName": "Preeti"
>> }
>>  },
>>  {
>> "_index": "investigations",
>> "_type": "TestCustom",
>> "_id": "6",
>> "_score": 1,
>> "_source": {
>>"name": "Preeti"
>> }
>>  }
>>   ]
>>}
>> }
>>  
>> Is this expected behavior?
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/79af9adb-5dce-4a7a-897b-16cd962c8a40%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/54ed8625-a8bc-4be4-b48a-18b364d51a66%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Preventing stop-of-the-world garbage collection

[joergpra...@gmail.com]

I'm interested in knowing more about G1 GC stability in Java 8, so I can
apply fixes to my production cluster, that is running stable for months
with G1 GC.

All I know are sporadic failures of Lucene 5 codec (which is under
development und not relaesed in ES) and a rare failure of a random junit
test on http://jenkins.elasticsearch.org (maybe a double free pointer) but
they seem not to be escalated into the OpenJDK issue tracker, so I can not
verify the cause, if it's G1 GC or not.

Jörg


On Tue, Dec 30, 2014 at 8:20 AM, Michal Taborsky 
wrote:

> Hi Christopher, thanks.
>
> Field and filter caches are not the problem, I think, they occupy only
> minority of the memory. The garbage collection in fact frees up a lot of
> memory, so I think the problem is that the standard GC that is supposed to
> run continuously cannot keep up. I will give G1 a try, though I have seen
> in several places that it's not recommended as it's not stable enough.
>
> Michal
>
> Dne úterý, 30. prosince 2014 1:55:57 UTC+1 Chris Rimondi napsal(a):
>>
>> +1 for using G1GC. In addition I would suggest not trying to fine tune GC
>> settings. If you have stop the world old GCs taking 20+ seconds you have a
>> more fundamental issue at play. I speak from experience on that. We had
>> similar issues and no amount of JVM/GC tuning could mask the fact we simply
>> didn't have enough memory.
>>
>> If you aren't already doing so look at the amount of heap used by the
>> filter and field caches. Are you capping them? If you aren't expensive
>> queries could saturate your entire heap. Along the same line keep tabs on
>> your evictions. ES provides granular metrics so you can look at both filter
>> and field cache evictions.
>>
>> On Mon, Dec 29, 2014 at 8:03 AM, joerg...@gmail.com 
>> wrote:
>>
>>> You said, very complex documents and queries, and 22 GB heap. Without
>>> knowing more about your queries and filters, it is hard to comment.  There
>>> is default query/filter caching in some cases.
>>>
>>> Jörg
>>>
>>> On Mon, Dec 29, 2014 at 1:55 PM, Michal Taborsky 
>>> wrote:
>>>
 Hi Jörg, thanks for your reply.

 What do you mean if we have setup caching? We do not have any special
 caching configuration, we use the defaults. How do you suggest we
 reconfigure ES? That is what I am trying to find out.

 All best,
 Michal


 Dne pondělí, 29. prosince 2014 12:06:43 UTC+1 Jörg Prante napsal(a):
>
> You could use G1 GC for nicer behavior regarding application stop
> times, but before tinkering with GC, it would be better to check if you
> have set up caching, and if it is possible to clear caches or reconfigure
> ES.
>
> Jörg
>
>
> On Mon, Dec 29, 2014 at 10:36 AM, Michal Taborsky <
> michal@gmail.com> wrote:
>
>> Hello everyone,
>>
>> we are using ES as a backend of an online service and occasionally,
>> we are hit by a big garbage collection, which stops the node completely 
>> and
>> causes all sorts of problems. The nodes have plenty of memory I think.
>> During the GC it looks like this.
>>
>> [cz-dc-v-313] [gc][young][2270193][2282693] duration [1.6m],
>> collections [3]/[2m], total [1.6m]/[17.6h], memory
>> [21.1gb]->[6.5gb]/[22gb], all_pools {[young] 
>> [478.6mb]->[224.7mb]/[599mb]}{[survivor]
>> [74.8mb]->[0b]/[74.8mb]}{[old] [20.6gb]->[6.3gb]/[21.3gb]}
>> [cz-dc-v-313] [gc][old][2270193][2344] duration [24.1s], collections
>> [1]/[2m], total [24.1s]/[6.1m], memory [21.1gb]->[6.5gb]/[22gb], 
>> all_pools
>> {[young] [478.6mb]->[224.7mb]/[599mb]}{[survivor]
>> [74.8mb]->[0b]/[74.8mb]}{[old] [20.6gb]->[6.3gb]/[21.3gb]}
>>
>> This might happen once a day, usually during a period of heavy
>> indexing, sometimes it doesn't. We tried decresing the heap size, but it
>> does not have that much of an effect. It makes the GC take a bit less 
>> time,
>> but makes it happen a bit more often.
>>
>> The data is actually fairly small in size, about 30G in total, but
>> very complex documents and queries. This is a 5-node cluster, the nodes
>> have 32G RAM with 22G assigned to ES heap.
>>
>> I know the manual says we should not touch the JVM GC settings but I
>> feel we might have to. Does anyone have any idea how to prevent these
>> garbage collections from ever happening?
>>
>> Thanks,
>> Michal
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to elasticsearc...@googlegroups.com.
>> To view this discussion on the web visit https://groups.google.com/d/
>> msgid/elasticsearch/29125088-8c43-4d97-b77b-71819fa11d09%40goo
>> glegroups.com
>> 

Re: copy_to and boost factor

[joergpra...@gmail.com]

Boosting is not supported in copy_to, see

https://github.com/elasticsearch/elasticsearch/pull/4972

Alternative, as quoted from the commit message

"Instead, users should either (1) query the common field AND the field that
requires boosting, or (2) the multi_match query will soon be able to
perform term-centric cross-field matching that will allow per-field
boosting at query time (coming in 1.1)."

See my multi_match gist for an example

https://gist.github.com/jprante/85126685973311603c8a

Jörg

On Tue, Dec 30, 2014 at 9:36 AM, Preeti Jain  wrote:

> Hi,
>
> Does the individual boost factor get honoured for fields getting copied
> into a custom _all field via copy_to option?
> I am facing issues with the behavior. Have a look at the example below
>
> PUT localhost:9200/investigations/TestCustom/_mapping
> {
>   "TestCustom": {
>   "properties": {
>   "custom_all":{
>   "type": "string"
>   },
>   "name":{
>   "type": "string",
>   "copy_to":"custom_all",
>   "boost":2.0
>   },
> "surName":{
>   "type": "string",
>   "copy_to":"custom_all"
>   },
>   "dateRecorded":{
>   "type": "date",
>   "copy_to":"custom_all"
>   },
> "Id":{
>   "type": "long",
>   "copy_to":"custom_all"
>   }
>   }
> }
> }
>
> Fiels "name" has been given a boost factor of 2.0
>
> I post 2 documents now
>
> POST localhost:9200/investigations/TestCustom/5
> {
> "surName":"Preeti"
> }
> POST localhost:9200/investigations/TestCustom/6
> {
> "name":"Preeti"
> }
>
> and then search the custom_all field
>
> GET localhost:9200/investigations/TestCustom/_search
> {
> "query":{
>"query_string": {
>"default_field": "custom_all",
>"query": "Preeti"
> }
> }
> }
>
> I expected document with id 6 to be the top result but the results were
>
> {
>"took": 40,
>"timed_out": false,
>"_shards": {
>   "total": 5,
>   "successful": 5,
>   "failed": 0
>},
>"hits": {
>   "total": 2,
>   "max_score": 1,
>   "hits": [
>  {
> "_index": "investigations",
> "_type": "TestCustom",
> "_id": "5",
> "_score": 1,
> "_source": {
>"surName": "Preeti"
> }
>  },
>  {
> "_index": "investigations",
> "_type": "TestCustom",
> "_id": "6",
> "_score": 1,
> "_source": {
>"name": "Preeti"
> }
>  }
>   ]
>}
> }
>
> Is this expected behavior?
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/79af9adb-5dce-4a7a-897b-16cd962c8a40%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFoMaubLqX%2BhODrq74mKtM33%2BAojT8NrDNO%2BpumtDHNCA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Different indexes for different Panels in Kibana

[Ashutosh Parab]

I want to configure Kibana in such a way that my different panels have 
different indexes. For example, histogram panel uses index 'X'' and table 
panel uses index 'Y'.
Is there any way to do this in Kibana 3. As far as I  checked, we can set 
Index only at Dashboard level and not at Panel level.
Please advice

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1405a67e-9572-4a16-b4ce-730b160f9d97%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

copy_to and boost factor

[Preeti Jain]

Hi,
 
Does the individual boost factor get honoured for fields getting copied 
into a custom _all field via copy_to option?
I am facing issues with the behavior. Have a look at the example below
 
PUT localhost:9200/investigations/TestCustom/_mapping
{
  "TestCustom": {
  "properties": { 
  "custom_all":{
  "type": "string"
  },
  "name":{
  "type": "string",
  "copy_to":"custom_all",
  "boost":2.0
  },
"surName":{
  "type": "string",
  "copy_to":"custom_all"
  },
  "dateRecorded":{
  "type": "date",
  "copy_to":"custom_all"
  },
"Id":{
  "type": "long",
  "copy_to":"custom_all"
  }
  }
}
}
 
Fiels "name" has been given a boost factor of 2.0
 
I post 2 documents now
 
POST localhost:9200/investigations/TestCustom/5
{
"surName":"Preeti"
}
POST localhost:9200/investigations/TestCustom/6
{
"name":"Preeti"
}
 
and then search the custom_all field
 
GET localhost:9200/investigations/TestCustom/_search
{
"query":{
   "query_string": {
   "default_field": "custom_all",
   "query": "Preeti"
}
}
}
 
I expected document with id 6 to be the top result but the results were
 
{
   "took": 40,
   "timed_out": false,
   "_shards": {
  "total": 5,
  "successful": 5,
  "failed": 0
   },
   "hits": {
  "total": 2,
  "max_score": 1,
  "hits": [
 {
"_index": "investigations",
"_type": "TestCustom",
"_id": "5",
"_score": 1,
"_source": {
   "surName": "Preeti"
}
 },
 {
"_index": "investigations",
"_type": "TestCustom",
"_id": "6",
"_score": 1,
"_source": {
   "name": "Preeti"
}
 }
  ]
   }
}
 
Is this expected behavior?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/79af9adb-5dce-4a7a-897b-16cd962c8a40%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ES upgrade 0.20.6 to 1.3.4 -> CorruptIndexException

[Georgeta Boanea]

Any ideas?

On Friday, December 19, 2014 11:40:37 AM UTC+1, Georgeta Boanea wrote:
>
> Hi All,
>
> After upgrading from ES 0.20.6 to 1.3.4 the following messages occurred:
>
> [2014-12-19 10:02:06.714 GMT] WARN || 
> elasticsearch[es-node-name][generic][T#14] 
> org.elasticsearch.cluster.action.shard  [es-node-name] [index-name][3] 
> sending failed shard for [index-name][3], node[qOTLmb3IQC2COXZh1n9O2w], 
> [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, 
> message [IndexShardGatewayRecoveryException[[index-name][3] failed to fetch 
> index version after copying it over]; nested: 
> CorruptIndexException[[index-name][3] Corrupted index 
> [corrupted_Ackui00SSBi8YXACZGNDkg] caused by: CorruptIndexException[did not 
> read all bytes from file: read 112 vs size 113 (resource: 
> BufferedChecksumIndexInput(NIOFSIndexInput(path="path/3/index/_uzm_2.del")))]];
>  
> ]]
>
> [2014-12-19 10:02:08.390 GMT] WARN || 
> elasticsearch[es-node-name][generic][T#20] 
> org.elasticsearch.indices.cluster  [es-node-name] [index-name][3] failed to 
> start shard
> org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: 
> [index-name][3] failed to fetch index version after copying it over
> at 
> org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:152)
> at 
> org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.lucene.index.CorruptIndexException: [index-name][3] 
> Corrupted index [corrupted_Ackui00SSBi8YXACZGNDkg] caused by: 
> CorruptIndexException[did not read all bytes from file: read 112 vs size 
> 113 (resource: 
> BufferedChecksumIndexInput(NIOFSIndexInput(path="path/3/index/_uzm_2.del")))]
> at org.elasticsearch.index.store.Store.failIfCorrupted(Store.java:353)
> at org.elasticsearch.index.store.Store.failIfCorrupted(Store.java:338)
> at 
> org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:119)
> ... 4 more
>
> Shard [3] of the index remains unallocated and the cluster remains in a 
> RED state.
>
> curl -XGET 'http://localhost:48012/_cluster/health?pretty=true'
> {
>   "cluster_name" : "cluster-name",
>   "status" : "red",
>   "timed_out" : false,
>   "number_of_nodes" : 5,
>   "number_of_data_nodes" : 5,
>   "active_primary_shards" : 10,
>   "active_shards" : 20,
>   "relocating_shards" : 0,
>   "initializing_shards" : 1,
>   "unassigned_shards" : 1
> }
>
> If I do an optimize (curl -XPOST 
> http://localhost:48012/index-name/_optimize?max_num_segments=1) for the 
> index before the update, everything is fine. Optimize works just before the 
> update, if is done after the update the problem remains the same.
>
> Any idea why this problem occurs?
> Is there another way to avoid this problem? I want to avoid optimize in 
> case of large volume of data.
>
> Thank you,
> Georgeta
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/9b5c6d6c-e8b5-4818-98d1-0ca64f289c5f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.