Re: Problem with "long" type field

[Igor Motov]

What I was trying to say is that the problem is with browsers parsing json. 
I am not really sure how you jumped to the conclusion that you cannot use 
Java API. Java API doesn't have any of these issues since it is using the 
same representation for long as the rest of elasticsearch. JavaScript on 
another hand is using a different internal representation of long numbers, 
so when you ask JavaScript to parse number -8848340816900692111 it loads it 
as -8848340816900692000. Try parsing your document using 
http://www.jsoneditoronline.org/ and you will see what I mean.

On Wednesday, 29 April 2015 02:24:18 UTC-4, Stabaoc wrote:
>
> So I can't use Java Api?
>
> 在 2015年4月29日星期三 UTC+8上午11:43:42，Igor Motov写道：
>>
>> It typically happens if you put data into elasticsearch using Sense or 
>> some other JavaScript-based applications. Large numbers like this one are 
>> getting rounded in JavaScript before they reach Elasticsearch. Please see 
>> https://github.com/elastic/elasticsearch/issues/5518#issuecomment-38540645 
>> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Felastic%2Felasticsearch%2Fissues%2F5518%23issuecomment-38540645&sa=D&sntz=1&usg=AFQjCNFznSzDQEOcZbgWn0lj7SzowuUfzw>
>>  
>> for more information about the issue.
>>
>> On Tuesday, 28 April 2015 23:02:16 UTC-4, Stabaoc wrote:
>>>
>>> I meet a problem . When I index an value , for example {"id": 
>>> -8848340816900692111},
>>> then i search it ,it shows that "id": -8848340816900692000. 
>>> Anyone can help? I want know why does elasticsearch do this and how can 
>>> i deal with.
>>> Thanks.
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/fa58ef0d-7a99-4840-9ac0-fd7f78dd44ca%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Problem with "long" type field

[Igor Motov]

It typically happens if you put data into elasticsearch using Sense or some 
other JavaScript-based applications. Large numbers like this one are 
getting rounded in JavaScript before they reach Elasticsearch. Please see 
https://github.com/elastic/elasticsearch/issues/5518#issuecomment-38540645 
for more information about the issue.

On Tuesday, 28 April 2015 23:02:16 UTC-4, Stabaoc wrote:
>
> I meet a problem . When I index an value , for example {"id": 
> -8848340816900692111},
> then i search it ,it shows that "id": -8848340816900692000. 
> Anyone can help? I want know why does elasticsearch do this and how can i 
> deal with.
> Thanks.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e12eb941-fe3c-43d3-931d-2efea9577525%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: exists filter broken on 1.5.0 with restored index?

[Igor Motov]

Thanks for checking. It's a bug, which should be fixed in 
1.5.1 https://github.com/elastic/elasticsearch/pull/10268

On Wednesday, 25 March 2015 13:43:28 UTC-4, Mads Martin Jørgensen wrote:
>
> They're similar. The 1.5.0 cluster has "created" : "1000199", and the 
> 1.4.1 cluster also has "created" : "1000199"
>
> On Wednesday, March 25, 2015 at 4:45:30 PM UTC+1, Igor Motov wrote:
>>
>> Hi Mads Martin,
>>
>> Could you check the version that is returned when you run curl 
>> "localhost:9200/my_own_index/_settings?pretty". The version will be in 
>>
>> "version" : {
>>   "created" : "XXX"
>> }
>>
>> Could you compare it to the version that is returned by the same index in 
>> the pre-1.5.0 cluster?
>>
>> Igor
>>
>> On Wednesday, 25 March 2015 09:27:03 UTC-4, Mads Martin Jørgensen wrote:
>>>
>>> Hello all,
>>>
>>> Just installed es-1.5.0 with cloud-aws-2.5.0 on a machine. Did a restore 
>>> of a snapshot made with es-1.4.1. All documents are there, but the exists 
>>> filter seems broken. The query that used to return all documents matching, 
>>> now return 0 documents, even though the field exists when reading the 
>>> documents.
>>>
>>> curl -XGET "http://localhost:9200/my_own_index/document/_search 
>>> <http://www.google.com/url?q=http%3A%2F%2Fcarter%3A9200%2Fskyfish_v2%2Fdocument%2F_search&sa=D&sntz=1&usg=AFQjCNHx9tllVr6Ip2n5zS9uz9FNweGcWg>"
>>>  
>>> -d'
>>>
>>> {
>>>
>>>"query": {
>>>
>>>   "constant_score": {
>>>
>>>  "filter": {
>>>
>>> "exists": {
>>>
>>>"field": "history"
>>>
>>> }
>>>
>>>  }
>>>
>>>   }
>>>
>>>}
>>>
>>> }'
>>>
>>>
>>> If we populate new documents, then the exists filter works just fine.
>>>
>>>
>>> Regards,
>>>
>>> Mads Martin
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/531c6e3c-5697-4fc0-b037-d8fbc438c5dd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: exists filter broken on 1.5.0 with restored index?

[Igor Motov]

Hi Mads Martin,

Could you check the version that is returned when you run curl 
"localhost:9200/my_own_index/_settings?pretty". The version will be in 

"version" : {
  "created" : "XXX"
}

Could you compare it to the version that is returned by the same index in 
the pre-1.5.0 cluster?

Igor

On Wednesday, 25 March 2015 09:27:03 UTC-4, Mads Martin Jørgensen wrote:
>
> Hello all,
>
> Just installed es-1.5.0 with cloud-aws-2.5.0 on a machine. Did a restore 
> of a snapshot made with es-1.4.1. All documents are there, but the exists 
> filter seems broken. The query that used to return all documents matching, 
> now return 0 documents, even though the field exists when reading the 
> documents.
>
> curl -XGET "http://localhost:9200/my_own_index/document/_search 
> "
>  
> -d'
>
> {
>
>"query": {
>
>   "constant_score": {
>
>  "filter": {
>
> "exists": {
>
>"field": "history"
>
> }
>
>  }
>
>   }
>
>}
>
> }'
>
>
> If we populate new documents, then the exists filter works just fine.
>
>
> Regards,
>
> Mads Martin
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/3937725f-6381-442b-9e2b-638913514d3b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: What is the best practice for periodic snapshotting with awc-cloud+s3

[Igor Motov]

Having too many snapshots is problematic. Each snapshot is done in 
incremental manner, so in order to figure out what changes and what is 
available all snapshots in the repository needs to be scanned, which takes 
time as number of snapshots growing. I would recommend pruning old 
snapshots as time goes by or starting snapshots into a new bucket/directory 
if you really need to maintain 2 hour resolution for 2 months old 
snapshots. The get command can sometimes hang because it's throttled by the 
on-going snapshot. 


On Wednesday, November 12, 2014 9:02:33 PM UTC-10, Sally Ahn wrote:
>
> I am also interested in this topic.
> We were snapshotting our cluster of two nodes every 2 hours (invoked via a 
> cron job) to an S3 repository (we were running ES 1.2.2 with 
> cloud-aws-plugin version 2.2.0, then we upgraded to ES 1.4.0 with 
> cloud-aws-plugin 2.4.0 but are still seeing issues described below).
> I've been seeing an increase in the time it takes to complete a snapshot 
> with each subsequent snapshot. 
> I see a thread 
> 
>  where 
> someone else was seeing the same thing, but that thread seems to have died.
> In my case, snapshots have gone from taking ~5 minutes to taking about an 
> hour, even between snapshots where data does not seem to have changed. 
>
> For example, you can see below a list of the snapshots stored in my S3 
> repo. Each snapshot is named with a timestamp of when my cron job invoked 
> the snapshot process. The S3 timestamp on the left shows the completion 
> time of that snapshot, and it's clear that it's steadily increasing:
>
> 2014-09-30 10:05   686   s3:///snapshot-2014.09.30-10:00:01
> 2014-09-30 12:05   686   s3:///snapshot-2014.09.30-12:00:01
> 2014-09-30 14:05   736   s3:///snapshot-2014.09.30-14:00:01
> 2014-09-30 16:05   736   s3:///snapshot-2014.09.30-16:00:01
> ...
> 2014-11-08 00:52  1488   s3:///snapshot-2014.11.08-00:00:01
> 2014-11-08 02:54  1488   s3:///snapshot-2014.11.08-02:00:01
> ...
> 2014-11-08 14:54  1488   s3:///snapshot-2014.11.08-14:00:01
> 2014-11-08 16:53  1488   s3:///snapshot-2014.11.08-16:00:01
> ...
> 2014-11-11 07:00  1638   s3:///snapshot-2014.11.11-06:00:01
> 2014-11-11 08:58  1638   s3:///snapshot-2014.11.11-08:00:01
> 2014-11-11 10:58  1638   s3:///snapshot-2014.11.11-10:00:01
> 2014-11-11 12:59  1638   s3:///snapshot-2014.11.11-12:00:01
> 2014-11-11 15:00  1638   s3:///snapshot-2014.11.11-14:00:01
> 2014-11-11 17:00  1638   s3:///snapshot-2014.11.11-16:00:01
>
> I suspected that this gradual increase was related to the accumulation of 
> old snapshots after I tested the following:
> 1. I created a brand new cluster with the same hardware specs in the same 
> datacenter and restored a snapshot of the problematic cluster taken few 
> days back (i.e. not the latest snapshot). 
> 2. I then backed up that restored data to a new empty bucket in the same 
> S3 region, and that was very fast...a minute or less. 
> 3. I then restored a later snapshot of the problematic cluster to the test 
> cluster and tried backing it up again to the new bucket, and that also took 
> about a minute or less.
>
> However, when I tried deleting the repository full of old snapshots from 
> the problematic cluster and registering a brand new empty bucket, I found 
> that my first snapshot to the new repository was also hanging indefinitely. 
> I finally had to kill my snapshot curl command. There were no errors in the 
> logs (the snapshot logger is very terse...wondering if anyone knows how to 
> increase the verbosity for it).
>
> So my theory seems to have been debunked, and I am again at a loss. I am 
> wondering whether the hanging snapshot is related to the slow snapshots I 
> was seeing before I deleted that old repository. I have seen several issues 
> in GitHub regarding hanging snapshots (#5958 
> , #7980 
> ) and have 
> tried using the elasticsearch-snapshot-cleanup 
>  utility on my 
> cluster both before and after I upgraded from version 1.2.2 to 1.4.0 (I 
> thought upgrading to 1.4.0 which included snapshot improvements may fix my 
> issues, but it did not), and the script is not finding any running 
> snapshots:
>
> [2014-11-13 05:37:45,451][INFO ][org.elasticsearch.node   ] [Golden 
> Archer] started
> [2014-11-13 05:37:45,451][INFO 
> ][org.elasticsearch.org.motovs.elasticsearch.snapshots.AbortedSnapshotCleaner]
>  
> No snapshots found
> [2014-11-13 05:37:45,452][INFO ][org.elasticsearch.node   ] [Golden 
> Archer] stopping ...
>
> Curling to _snapshot/REPO/_status also returns no ongoing snapshots:
>
> curl -XGET 
> 'http://:9200/_snapshot/s3_backup_repo/_status?pretty=true'
> {
>   "snapshots" : [ 

Re: Snapshots don't get compressed

[Igor Motov]

There were two reasons for not enabling compression on data files. First of 
all, the way "chunking" in snapshot/restore API was implemented didn't 
allow simple implementation of compression on data files. Moreover, the 
data files are already compressed to a certain degree. In my tests I was 
getting about 20% compressions rates on index data with recent versions of 
Elasticsearch (we have a limitation that we can compress only one file at a 
time). So, difficulties with implementation together with limit benefits 
made this feature not very compelling. After recent refactoring of the 
storage code, it's now much easier to add this feature in if it makes 
sense. However, I am really curious how you've got such great compression 
rates. Which version of Elasticsearch were these indices created with? Did 
you upgrade from older version of elasticsearch recently and most of your 
data files are created with older versions of Lucene?

On Thursday, September 18, 2014 10:53:58 AM UTC-4, Domonkos Tomcsanyi wrote:
>
>  thank you again, I'll keep an eye on this issue.
>
> Domonkos
>
> 2014.09.18. 16:38 keltezéssel, David Pilato írta:
>  
> I don't know. I think this could happen in the future but unsure though.
> May be Igor could answer this?
>
>  Here is a related doc PR: 
> https://github.com/elasticsearch/elasticsearch/pull/7654 
> 
>
>  
>  -- 
> *David Pilato* | *Technical Advocate* | *Elasticsearch.com* 
> @dadoonet  | @elasticsearchfr 
> 
>
>
> Le 18 septembre 2014 à 16:35:14, Tomcsányi, Domonkos (tomcsan...@modit.hu) 
> a écrit:
>
>  Thank you for the answer, but may I know why? What is the reason behind 
> this? 
>
> thanks, 
> Domonkos 
>
> 2014.09.18. 16:15 keltezéssel, David Pilato írta: 
> > Only metadata are compressed. 
> > 
> > -- 
> > David ;-) 
> > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs 
> > 
> >> Le 18 sept. 2014 à 15:36, "Tomcsányi, Domonkos"  
>  a écrit : 
> >> 
> >> Hi Everyone, 
> >> 
> >> I just started experimenting with the cool snapshot feature of ES 
> (using ES 1.3.2 on Ubuntu 14.04) using curator. I created a new repository 
> on a mounted NFS storage, using only the default options (compression 
> turned on). I checked it using curl: 
> >> 
> >> user@myserver:~# curl -XGET '
> http://IP_ADDRESS:9200/_snapshot/logBack?pretty' 
> >> { 
> >> "logBack" : { 
> >> "type" : "fs", 
> >> "settings" : { 
> >> "compress" : "true", 
> >> "location" : "/es_snapshots" 
> >> } 
> >> } 
> >> } 
> >> 
> >> So, after that I used curator to create a snapshot of some older 
> indices. The process finished after some minutes, so I decided to have a 
> look at the files it created. It turned out that the snapshot's files take 
> up exactly as much space as the indices did originally while they were in 
> the cluster, so no compression happened at all. This is kind of a problem 
> for me, because I assumed that compression will greatly reduce the size of 
> the indices I put in a snapshot. So is there anything I'm doing wrong? 
> >> 
> >> Thank you, 
> >> Domonkos 
> >> 
> >> -- 
> >> You received this message because you are subscribed to the Google 
> Groups "elasticsearch" group. 
> >> To unsubscribe from this group and stop receiving emails from it, send 
> an email to elasticsearch+unsubscr...@googlegroups.com. 
> >> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/541ADFEF.7080800%40modit.hu
> .
> >> For more options, visit https://groups.google.com/d/optout. 
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group. 
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearch+unsubscr...@googlegroups.com. 
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/541AED9F.4030007%40modit.hu
> .
> For more options, visit https://groups.google.com/d/optout. 
>  
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/etPan.541aee74.625558ec.b066%40MacBook-Air-de-David.local
>  
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
>
> 

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.c

Re: Data loss after network disconnect

[Igor Motov]

How were these nodes doing in terms of available heap space before the 
disconnects occurred? 

On Wednesday, September 10, 2014 6:26:19 AM UTC-4, Israel Tsadok wrote:
>
> A temporary network disconnect of the master node caused a torrent of 
> RELOCATING shards, and then one shard remained UNASSIGNED and the cluster 
> state was left red.
>
> looking inside the index directory for the shard on the disk, I found that 
> it was empty (i.e., the _state and translog dirs were there, but the index 
> dir had no files).
>
> Looking at the log files, I see that the disconnect happened around 
> 11:42:05, and a few minutes later I start seeing these error messages:
>
> *[2014-09-10 11:45:33,341]*[WARN ][indices.cluster  ] 
> [buzzilla_data008] [el-2011-10-31-][0] failed to start shard
> *[2014-09-10 11:45:33,342]*[WARN ][cluster.action.shard ] 
> [buzzilla_data008] [el-2011-10-31-][0] sending failed shard for 
> [el-2011-10-31-][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], 
> s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message 
> [IndexShardGatewayRecoveryException[[el-2011-10-31-][0] failed to fetch 
> index version after copying it over]; nested: 
> IndexShardGatewayRecoveryException[[el-2011-10-31-][0] shard allocated 
> for local recovery (post api), should exist, but doesn't, current files: 
> []]; nested: IndexNotFoundException[no segments* file found in 
> store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-/0/index),
>  
> type=MERGE, rate=20.0)]): files: []]; ]]
>
> The relevant log files are at 
> https://gist.github.com/itsadok/97453743d6b211681aca
> data009 is the original master, data017 is the new master, and data008 is 
> where I found the empty index directory.
>
> I had to delete the unassigned index from the cluster to return to green 
> state.
> I am running Elasticsearch 1.2.1 in a 20 node cluster. 
>
> How does this happen? What can I do to prevent this from happening again?
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/749729f6-daa1-470c-a835-d8f5dd85ad87%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Snapshot compress not compressing?

[Igor Motov]

At the moment, compression is applied only to metadata files (index mapping 
and settings basically). Data files are not compressed. 

On Monday, September 8, 2014 5:22:09 AM UTC-4, Russell Seymour wrote:
>
> Good morning,
>
> I experienced the exact same issue on Friday as well.
>
> I have an Elastic Search cluster (1.3.2) running on Windows using Oracle 
> Java 1.7.0_67.  We needed a backup strategy and purposely upgraded to this 
> version to take advantage of the snapshots feature.
> The size of the indexes in the cluster is about 40Gb and even with the 
> 'compress' option explicitly set to true (as in Paul's post and in the 
> documentation) the snapshot is about 40Gb,
>
> Is there a work around to get this working or some other fix?
>
> Thanks, Russell
>
> On Friday, 5 September 2014 22:27:09 UTC+1, ppearcy wrote:
>>
>> I am playing around with snapshot/restore and have a local 1.3.2 cluster 
>> running on Mac OS X with 894MB of index data. 
>>
>> I have registered a backup repository like so (straight from the docs):
>>
>> curl -XPUT 'http://localhost:9200/_snapshot/my_backup' -d '{
>> "type": "fs",
>> "settings": {
>> "location": "/tmp/backups/my_backup",
>> "compress": true
>> }
>> }'
>>
>> Then run the snapshot (again straight from the docs):
>>
>> curl -XPUT 
>> "localhost:9200/_snapshot/my_backup/snapshot_1?wait_for_completion=true"
>>
>> The snapshot runs fine, but the backup directory that is generated is 
>> 890MB, which tells me that compression isn't kicking in. When I set 
>> compress: false, I get the same results. 
>>
>> If I tar/gz that directory it gets squashed down to 204MB. I'd expect the 
>> compressed snapshot from ES to be somewhere in that ballpark.
>>
>> Am I doing something wrong or is there a bug?
>> Thanks and Best Regards,
>> Paul
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/ab00335a-8216-45df-ab8e-09b33086ecaf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: does snapshot restore lead to a memory leak?

[Igor Motov]

So, you are running out of threads not memory. Are you re-registering 
repository every time you restore from it? If you do, you might be running 
into this issue https://github.com/elasticsearch/elasticsearch/issues/6181

On Thursday, July 3, 2014 2:06:38 PM UTC-4, JoeZ99 wrote:
>
> Igor.
> I'm posting a pdf document with some graphs I think are quite enlightening 
> . The "jvm threads" is particularly interesting. 
> the times are utc-4. and during the jvm growing period is when most of the 
> restore process have been taking place.
> Igor, I will send you the reports you asked me in an email, since they 
> contain filesystem data. Hope you don't mind. 
>
> The graphs contain data from two elasticsearch clusters. ES1 is the one 
> we've  been talking about in this thread. ES4 is on cluster devoted to two 
> indices, not very big but with a highly search demand.
>
>
> txs!!!
>
> On Monday, June 30, 2014 8:53:10 AM UTC-4, JoeZ99 wrote:
>>
>> We have one one-machine cluster with about 1k indices. It used to work 
>> flawlessly , (being with a high load, of course)
>>
>> but since we started to use heavily the snapshot-restore feature, it's 
>> getting its memory exhausted within 7 days of use. The cluster make about 
>> 700 restore proceedings during the day. Maybe there are some memory 
>> considerations when using the restore feature???
>>
>> -- 
>> uh, oh .
>>
>> 
>>  
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e6705c17-ea08-48fa-873c-b44dc797a9d4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: does snapshot restore lead to a memory leak?

[Igor Motov]

So, your "search-only" machines are running out of memory, while your 
"index-only" machines are doing fine. Did I understand you correctly? Could 
you send me nodes stats (curl "localhost:9200/_nodes/stats?pretty") from 
the machine that runs out of memory, please run stats a few times with 1 
hour interval. I would like to see how memory consumption is increasing 
over time. Please, also run nodes info ones (curl "localhost:9200/_nodes") 
and post here (or send me by email) the results. Thanks!

On Wednesday, July 2, 2014 10:15:46 AM UTC-4, JoeZ99 wrote:
>
> Hey, Igor, thanks for answering! and sorry for the delay. Didn't catch the 
> update.
>
> I explain:
>
>- we have one cluster of one machine which is only meant for serving 
> search requests. the goal is  not to index anything to it. It contains 1.7k 
> indices, give it or take it. 
>- every day, those 1.7k indices are reindexed, and snapshoted in pairs 
> to a S3 repository (producint 850 snapshots)repository. 
>- every day, the one "reading only" cluster of the first point restores 
> those 850 snapshots to "update" its 1.7k indices from that same S3 
> repository. 
>
> It works like a real charm. Load has dropped dramatically, and we can set 
> a "farm" of temporary machines to do the indexing duties. 
>
> But memory consumption never stops growing.
>
> we don't get any "out of memory" error or anything. In fact, there is 
> nothing in the logs that shows any error, but after a week or a few days, 
> the host has its memory almost exhausted and elasticsearch is not 
> responding. The memory consumption is of course way ahead of the HEAP_SIZE
> We have to restart it and, when we do it we get the following error:
>
> java.util.concurrent.RejectedExecutionException: Worker has already been 
> shutdown
> at org.elasticsearch.common.netty.channel.socket.nio.
> AbstractNioSelector.registerTask(AbstractNioSelector.java:120)
> at org.elasticsearch.common.netty.channel.socket.nio.
> AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72)
> at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.
> executeInIoThread(NioWorker.java:36)
> at org.elasticsearch.common.netty.channel.socket.nio.
> AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:56)
> at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.
> executeInIoThread(NioWorker.java:36)
> at org.elasticsearch.common.netty.channel.socket.nio.
> AbstractNioChannelSink.execute(AbstractNioChannelSink.java:34)
> at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.
> execute(DefaultChannelPipeline.java:636)
> at org.elasticsearch.common.netty.channel.Channels.
> fireExceptionCaughtLater(Channels.java:496)
> at org.elasticsearch.common.netty.channel.AbstractChannelSink.
> exceptionCaught(AbstractChannelSink.java:46)
> at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.
> notifyHandlerException(DefaultChannelPipeline.java:658)
> at org.elasticsearch.common.netty.channel.
> DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(
> DefaultChannelPipeline.java:781)
> at org.elasticsearch.common.netty.channel.Channels.write(Channels.
> java:725)
> at org.elasticsearch.common.netty.handler.codec.oneone.
> OneToOneEncoder.doEncode(OneToOneEncoder.java:71)
> at org.elasticsearch.common.netty.handler.codec.oneone.
> OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
> at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.
> sendDownstream(DefaultChannelPipeline.java:591)
> at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.
> sendDownstream(DefaultChannelPipeline.java:582)
> at org.elasticsearch.common.netty.channel.Channels.write(Channels.
> java:704)
> at org.elasticsearch.common.netty.channel.Channels.write(Channels.
> java:671)
> at org.elasticsearch.common.netty.channel.AbstractChannel.write(
> AbstractChannel.java:248)
> at org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(
> NettyHttpChannel.java:158)
> at org.elasticsearch.rest.action.search.RestSearchAction$1.
> onResponse(RestSearchAction.java:106)
> at org.elasticsearch.rest.action.search.RestSearchAction$1.
> onResponse(RestSearchAction.java:98)
> at org.elasticsearch.action.search.type.
> TransportSearchQueryAndFetchAction$AsyncAction.innerFinishHim(
> TransportSearchQueryAndFetchAction.java:94)
> at org.elasticsearch.action.search.type.
> TransportSearchQueryAndFetchAction$AsyncAction.moveToSecondPhase(
> TransportSearchQueryAndFetchAction.java:77)
> at org.elasticsearch.action.search.type.
> TransportSearchTypeAction$BaseAsyncAction.innerMoveToSecondPhase(
> TransportSearchTypeAction.java:425)
> at org.elasticsearch.action.search.type.
> TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(
> TransportSearchTypeAction.jav

Re: does snapshot restore lead to a memory leak?

[Igor Motov]

Just to make sure I got it right, you really meant 700 restores (not just 
700 snapshots), correct? What type of repository are you using? Could you 
add a bit more details about your use case?

On Monday, June 30, 2014 8:53:10 AM UTC-4, JoeZ99 wrote:
>
> We have one one-machine cluster with about 1k indices. It used to work 
> flawlessly , (being with a high load, of course)
>
> but since we started to use heavily the snapshot-restore feature, it's 
> getting its memory exhausted within 7 days of use. The cluster make about 
> 700 restore proceedings during the day. Maybe there are some memory 
> considerations when using the restore feature???
>
> -- 
> uh, oh .
>
> 
>  

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/6158eb50-bdbd-40c3-80fb-b18102cacb6d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Can't stop a snapshot running on my cluster

[Igor Motov]

Yes, by "full cluster restart" I meant shutting down all nodes and then 
starting them up again, which means downtime. However, after thinking about 
the issue over the long weekend, I wrote a simple utility that cleans up 
snapshots without need to restart the cluster 
- https://github.com/imotov/elasticsearch-snapshot-cleanup 

On Saturday, May 24, 2014 2:53:27 PM UTC-4, Andrew Vos wrote:
>
> Well it's the only way I can do it without downtime. Unless of course by 
> "full cluster restart" you mean restarting one node at a time?
>
>
> On Sat, May 24, 2014 at 7:51 PM, Igor Motov  wrote:
>
>> If your cluster is setup correctly (with proper value set 
>> for discovery.zen.minimum_master_nodes) they shouldn't. But if you are 
>> running without discovery.zen.minimum_master_nodes set, they might indeed 
>> form a new cluster. Obviously some shards might end up in one cluster and 
>> not in the other and if you are indexing while this is happening you will 
>> lose some data. I would say it's pretty. extreme way to test full 
>> cluster restart. 
>>
>>
>> On Saturday, May 24, 2014 2:38:44 PM UTC-4, Andrew Vos wrote:
>>>
>>> Ok. While you're here, one other question I would like answered:
>>>
>>> I have 10 nodes in a cluster. I want to break out three nodes into a 
>>> different cluster as a kind of backup to test out this full cluster 
>>> restart. Would it be safe to just block the other three nodes from 
>>> connecting to the main cluster? Would they form their own?
>>>
>>>
>>> On Sat, May 24, 2014 at 7:35 PM, Igor Motov  wrote:
>>>
>>>> It was caused by this bug - https://github.com/
>>>> elasticsearch/elasticsearch/issues/5958 The only recovery option right 
>>>> now is full cluster restart. 
>>>>
>>>> On Saturday, May 24, 2014 2:30:06 PM UTC-4, Andrew Vos wrote:
>>>>>
>>>>> Right ok here's the cluster state https://gist.github.com/
>>>>> AndrewVos/29de3c6735bbd7808a81
>>>>>
>>>>>
>>>>> On Sat, May 24, 2014 at 7:18 PM, Andrew Vos wrote:
>>>>>
>>>>>> 1.0.0. What do you mean by state exactly?
>>>>>>
>>>>>>
>>>>>> On Sat, May 24, 2014 at 6:33 PM, Igor Motov  wrote:
>>>>>>
>>>>>>> Which version of elsticsearch are you using? Can you send me the 
>>>>>>> current cluster state?
>>>>>>>
>>>>>>> On Saturday, May 24, 2014 10:17:43 AM UTC-4, Andrew Vos wrote:
>>>>>>>>
>>>>>>>> A few days ago I started a snapshot, but instead of using a shared 
>>>>>>>> network I used the local filesystem. Because my root partition only 
>>>>>>>> had 8gb 
>>>>>>>> (and this is where I stored the snapshots) the partition got filled up 
>>>>>>>> and 
>>>>>>>> three of my seven elasticsearch boxes crashed almost instantly.
>>>>>>>>
>>>>>>>> I've since created a new cluster and let the data replicate over. 
>>>>>>>> The problem now is, I can't seem to cancel this snapshot!
>>>>>>>>
>>>>>>>> Looking at the snapshot:
>>>>>>>> *curl -XGET "localhost:/_snapshot/production_backup/_snapshot1"*
>>>>>>>> *{"error":"RemoteTransportException[[Smuggler][inet[/172.17.0.2 
>>>>>>>> <http://172.17.0.2>:9300]][cluster/snapshot/get]]; nested: 
>>>>>>>> SnapshotMissingException[[production_backup:_snapshot1] is missing]; 
>>>>>>>> nested: FileNotFoundException[/ebs/snapshot-backup/snapshot-_snapshot1 
>>>>>>>> (No 
>>>>>>>> such file or directory)]; ","status":404}%   *
>>>>>>>>
>>>>>>>> Starting a new snapshot:
>>>>>>>> *curl -XPUT 
>>>>>>>> "localhost:/_snapshot/production_backup/snapshot_1?wait_for_completion=false"*
>>>>>>>> *{"error":"RemoteTransportException[[Smuggler][inet[/172.17.0.2 
>>>>>>>> <http://172.17.0.2>:9300]][cluster/snapshot/create]]; nested: 
>>>>>>>> ConcurrentSnapshotExecutionException[[production_backup:snapshot_1] a 
>>>>>>>> snapshot is a

Re: Can't stop a snapshot running on my cluster

[Igor Motov]

If your cluster is setup correctly (with proper value set 
for discovery.zen.minimum_master_nodes) they shouldn't. But if you are 
running without discovery.zen.minimum_master_nodes set, they might indeed 
form a new cluster. Obviously some shards might end up in one cluster and 
not in the other and if you are indexing while this is happening you will 
lose some data. I would say it's pretty. extreme way to test full 
cluster restart. 

On Saturday, May 24, 2014 2:38:44 PM UTC-4, Andrew Vos wrote:
>
> Ok. While you're here, one other question I would like answered:
>
> I have 10 nodes in a cluster. I want to break out three nodes into a 
> different cluster as a kind of backup to test out this full cluster 
> restart. Would it be safe to just block the other three nodes from 
> connecting to the main cluster? Would they form their own?
>
>
> On Sat, May 24, 2014 at 7:35 PM, Igor Motov  wrote:
>
>> It was caused by this bug - 
>> https://github.com/elasticsearch/elasticsearch/issues/5958 The only 
>> recovery option right now is full cluster restart. 
>>
>> On Saturday, May 24, 2014 2:30:06 PM UTC-4, Andrew Vos wrote:
>>>
>>> Right ok here's the cluster state https://gist.github.com/
>>> AndrewVos/29de3c6735bbd7808a81
>>>
>>>
>>> On Sat, May 24, 2014 at 7:18 PM, Andrew Vos wrote:
>>>
>>>> 1.0.0. What do you mean by state exactly?
>>>>
>>>>
>>>> On Sat, May 24, 2014 at 6:33 PM, Igor Motov  wrote:
>>>>
>>>>> Which version of elsticsearch are you using? Can you send me the 
>>>>> current cluster state?
>>>>>
>>>>> On Saturday, May 24, 2014 10:17:43 AM UTC-4, Andrew Vos wrote:
>>>>>>
>>>>>> A few days ago I started a snapshot, but instead of using a shared 
>>>>>> network I used the local filesystem. Because my root partition only had 
>>>>>> 8gb 
>>>>>> (and this is where I stored the snapshots) the partition got filled up 
>>>>>> and 
>>>>>> three of my seven elasticsearch boxes crashed almost instantly.
>>>>>>
>>>>>> I've since created a new cluster and let the data replicate over. The 
>>>>>> problem now is, I can't seem to cancel this snapshot!
>>>>>>
>>>>>> Looking at the snapshot:
>>>>>> *curl -XGET "localhost:/_snapshot/production_backup/_snapshot1"*
>>>>>> *{"error":"RemoteTransportException[[Smuggler][inet[/172.17.0.2 
>>>>>> <http://172.17.0.2>:9300]][cluster/snapshot/get]]; nested: 
>>>>>> SnapshotMissingException[[production_backup:_snapshot1] is missing]; 
>>>>>> nested: FileNotFoundException[/ebs/snapshot-backup/snapshot-_snapshot1 
>>>>>> (No 
>>>>>> such file or directory)]; ","status":404}%   *
>>>>>>
>>>>>> Starting a new snapshot:
>>>>>> *curl -XPUT 
>>>>>> "localhost:/_snapshot/production_backup/snapshot_1?wait_for_completion=false"*
>>>>>> *{"error":"RemoteTransportException[[Smuggler][inet[/172.17.0.2 
>>>>>> <http://172.17.0.2>:9300]][cluster/snapshot/create]]; nested: 
>>>>>> ConcurrentSnapshotExecutionException[[production_backup:snapshot_1] a 
>>>>>> snapshot is already running]; ","status":503}% *
>>>>>>
>>>>>> This just never completes:
>>>>>>
>>>>>> curl -XDELETE "localhost:/_snapshot/production_backup/snapshot_1"
>>>>>>
>>>>>>
>>>>>> Any idea how I can solve this?
>>>>>>
>>>>>  -- 
>>>>> You received this message because you are subscribed to a topic in the 
>>>>> Google Groups "elasticsearch" group.
>>>>> To unsubscribe from this topic, visit https://groups.google.com/d/
>>>>> topic/elasticsearch/rtffJxkKyzg/unsubscribe.
>>>>> To unsubscribe from this group and all its topics, send an email to 
>>>>> elasticsearch+unsubscr...@googlegroups.com.
>>>>> To view this discussion on the web visit https://groups.google.com/d/
>>>>> msgid/elasticsearch/41136f07-3afd-4452-87e4-a54f983db539%
>>>>> 40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/41136f07-3afd-4452-87e4-a54f983db539%40googlegroups.co

Re: Can't stop a snapshot running on my cluster

[Igor Motov]

It was caused by this bug 
- https://github.com/elasticsearch/elasticsearch/issues/5958 The only 
recovery option right now is full cluster restart. 

On Saturday, May 24, 2014 2:30:06 PM UTC-4, Andrew Vos wrote:
>
> Right ok here's the cluster state 
> https://gist.github.com/AndrewVos/29de3c6735bbd7808a81
>
>
> On Sat, May 24, 2014 at 7:18 PM, Andrew Vos  wrote:
>
>> 1.0.0. What do you mean by state exactly?
>>
>>
>> On Sat, May 24, 2014 at 6:33 PM, Igor Motov  wrote:
>>
>>> Which version of elsticsearch are you using? Can you send me the current 
>>> cluster state?
>>>
>>> On Saturday, May 24, 2014 10:17:43 AM UTC-4, Andrew Vos wrote:
>>>>
>>>> A few days ago I started a snapshot, but instead of using a shared 
>>>> network I used the local filesystem. Because my root partition only had 
>>>> 8gb 
>>>> (and this is where I stored the snapshots) the partition got filled up and 
>>>> three of my seven elasticsearch boxes crashed almost instantly.
>>>>
>>>> I've since created a new cluster and let the data replicate over. The 
>>>> problem now is, I can't seem to cancel this snapshot!
>>>>
>>>> Looking at the snapshot:
>>>> *curl -XGET "localhost:/_snapshot/production_backup/_snapshot1"*
>>>> *{"error":"RemoteTransportException[[Smuggler][inet[/172.17.0.2 
>>>> <http://172.17.0.2>:9300]][cluster/snapshot/get]]; nested: 
>>>> SnapshotMissingException[[production_backup:_snapshot1] is missing]; 
>>>> nested: FileNotFoundException[/ebs/snapshot-backup/snapshot-_snapshot1 (No 
>>>> such file or directory)]; ","status":404}%   *
>>>>
>>>> Starting a new snapshot:
>>>> *curl -XPUT 
>>>> "localhost:/_snapshot/production_backup/snapshot_1?wait_for_completion=false"*
>>>> *{"error":"RemoteTransportException[[Smuggler][inet[/172.17.0.2 
>>>> <http://172.17.0.2>:9300]][cluster/snapshot/create]]; nested: 
>>>> ConcurrentSnapshotExecutionException[[production_backup:snapshot_1] a 
>>>> snapshot is already running]; ","status":503}% *
>>>>
>>>> This just never completes:
>>>>
>>>> curl -XDELETE "localhost:/_snapshot/production_backup/snapshot_1"
>>>>
>>>>
>>>> Any idea how I can solve this?
>>>>
>>>  -- 
>>> You received this message because you are subscribed to a topic in the 
>>> Google Groups "elasticsearch" group.
>>> To unsubscribe from this topic, visit 
>>> https://groups.google.com/d/topic/elasticsearch/rtffJxkKyzg/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to 
>>> elasticsearch+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/elasticsearch/41136f07-3afd-4452-87e4-a54f983db539%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/41136f07-3afd-4452-87e4-a54f983db539%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/9573402c-d7fe-419d-866f-cff1196fcfc4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Can't stop a snapshot running on my cluster

[Igor Motov]

I meant the output of the cluster state 
<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-state.html#cluster-state>
command:

curl -XGET 'http://localhost:9200/_cluster/state'

It might be large and will contain information about your cluster that you 
might not want to share publicly (index mappings). If this is the case, 
please feel free to send it to me by email.

Igor


On Saturday, May 24, 2014 2:18:14 PM UTC-4, Andrew Vos wrote:
>
> 1.0.0. What do you mean by state exactly?
>
>
> On Sat, May 24, 2014 at 6:33 PM, Igor Motov  wrote:
>
>> Which version of elsticsearch are you using? Can you send me the current 
>> cluster state?
>>
>> On Saturday, May 24, 2014 10:17:43 AM UTC-4, Andrew Vos wrote:
>>>
>>> A few days ago I started a snapshot, but instead of using a shared 
>>> network I used the local filesystem. Because my root partition only had 8gb 
>>> (and this is where I stored the snapshots) the partition got filled up and 
>>> three of my seven elasticsearch boxes crashed almost instantly.
>>>
>>> I've since created a new cluster and let the data replicate over. The 
>>> problem now is, I can't seem to cancel this snapshot!
>>>
>>> Looking at the snapshot:
>>> *curl -XGET "localhost:/_snapshot/production_backup/_snapshot1"*
>>> *{"error":"RemoteTransportException[[Smuggler][inet[/172.17.0.2 
>>> <http://172.17.0.2>:9300]][cluster/snapshot/get]]; nested: 
>>> SnapshotMissingException[[production_backup:_snapshot1] is missing]; 
>>> nested: FileNotFoundException[/ebs/snapshot-backup/snapshot-_snapshot1 (No 
>>> such file or directory)]; ","status":404}%   *
>>>
>>> Starting a new snapshot:
>>> *curl -XPUT 
>>> "localhost:/_snapshot/production_backup/snapshot_1?wait_for_completion=false"*
>>> *{"error":"RemoteTransportException[[Smuggler][inet[/172.17.0.2 
>>> <http://172.17.0.2>:9300]][cluster/snapshot/create]]; nested: 
>>> ConcurrentSnapshotExecutionException[[production_backup:snapshot_1] a 
>>> snapshot is already running]; ","status":503}% *
>>>
>>> This just never completes:
>>>
>>> curl -XDELETE "localhost:/_snapshot/production_backup/snapshot_1"
>>>
>>>
>>> Any idea how I can solve this?
>>>
>>  -- 
>> You received this message because you are subscribed to a topic in the 
>> Google Groups "elasticsearch" group.
>> To unsubscribe from this topic, visit 
>> https://groups.google.com/d/topic/elasticsearch/rtffJxkKyzg/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to 
>> elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/41136f07-3afd-4452-87e4-a54f983db539%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/41136f07-3afd-4452-87e4-a54f983db539%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e292c3d2-b0bf-438a-986b-e32db3f2dd7a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Can't stop a snapshot running on my cluster

[Igor Motov]

Which version of elsticsearch are you using? Can you send me the current 
cluster state?

On Saturday, May 24, 2014 10:17:43 AM UTC-4, Andrew Vos wrote:
>
> A few days ago I started a snapshot, but instead of using a shared network 
> I used the local filesystem. Because my root partition only had 8gb (and 
> this is where I stored the snapshots) the partition got filled up and three 
> of my seven elasticsearch boxes crashed almost instantly.
>
> I've since created a new cluster and let the data replicate over. The 
> problem now is, I can't seem to cancel this snapshot!
>
> Looking at the snapshot:
> *curl -XGET "localhost:/_snapshot/production_backup/_snapshot1"*
> *{"error":"RemoteTransportException[[Smuggler][inet[/172.17.0.2:9300]][cluster/snapshot/get]];
>  
> nested: SnapshotMissingException[[production_backup:_snapshot1] is 
> missing]; nested: 
> FileNotFoundException[/ebs/snapshot-backup/snapshot-_snapshot1 (No such 
> file or directory)]; ","status":404}%   *
>
> Starting a new snapshot:
> *curl -XPUT 
> "localhost:/_snapshot/production_backup/snapshot_1?wait_for_completion=false"*
> *{"error":"RemoteTransportException[[Smuggler][inet[/172.17.0.2:9300]][cluster/snapshot/create]];
>  
> nested: ConcurrentSnapshotExecutionException[[production_backup:snapshot_1] 
> a snapshot is already running]; ","status":503}% *
>
> This just never completes:
>
> curl -XDELETE "localhost:/_snapshot/production_backup/snapshot_1"
>
>
> Any idea how I can solve this?
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/41136f07-3afd-4452-87e4-a54f983db539%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Snapshot Duration increasing over time

[Igor Motov]

Hi Dipesh,

I have a few questions. Are you still on S3? Which version of elasticsearch 
are you using? How many snapshots do you currently keep in S3? How fast is 
your index growing over time?

Igor

On Wednesday, May 7, 2014 6:58:05 AM UTC-4, Dipesh Patel wrote:
>
> Hi
>
> We've noticed recently that our snapshot durations are increasing over 
> time. Our rate of flow of data going into elasticsearch has remained fairly 
> constant. Though we do create new indices everyday ( though this is a fixed 
> number that doesn't vary from day to day ). We are currently snapshoting, 
> or trying to snapshot every hour. However with the snapshots taking a 
> progressively longer time this is proving difficult.
>
> Here's some stats showing our time to finish:
>
> Name   
> Duration ( milli )
>   snapshot_2014-05-01_01:30:00 4497010  snapshot_2014-05-01_03:30:00 
> 4513037  snapshot_2014-05-01_05:30:00 4770288  
> snapshot_2014-05-01_07:30:00 5413361  snapshot_2014-05-01_11:30:00 6978384  
> snapshot_2014-05-01_13:30:00 6907554  snapshot_2014-05-01_15:30:00 7388500  
> This is just the tail end originally the snapshots were only taking 7-8 
> mins to run, they've just been getting progressively longer.
>
> Any help to debug etc appreciated.
>
> Dip
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/96ca5fa6-f366-4f9e-b3c1-a57b511a341e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: transport.tcp.port doesn't work for localhost?

[Igor Motov]

You should specify the same cluster name for both node and transport 
client. It looks like they are running in different clusters:

[2014-04-02 15:19:23,262][WARN ][org.elasticsearch.client.transport] [Humus 
Sapien] node [#transport#-1][d][inet[localhost/127.0.0.1:9300]] not part of 
the cluster Cluster [elasticsearch], ignoring...

On Wednesday, April 2, 2014 10:27:19 AM UTC-4, Dario Rossi wrote:
>
> I forgot, after setting up the embedded node, I wait the cluster status to 
> be Yellow with
>
>  Client client = node.client();
> client.admin().cluster().prepareHealth().setWaitForYellowStatus().
> execute().actionGet();
>
>
> this is done on the embedded node.
>
> Il giorno mercoledì 2 aprile 2014 15:24:35 UTC+1, Dario Rossi ha scritto:
>>
>> I set it to local = false and now I don't get anymore the connection 
>> refused. But unfortunaly I get another thing:
>>
>>
>> [2014-04-02 15:19:23,218][DEBUG][org.elasticsearch.transport.netty] [
>> Humus Sapien] using worker_count[12], port[9300-9400], 
>> bind_host[null],publish_host
>> [null], compress[false], connect_timeout[30s], connections_per_node[2/3/6
>> /1/1], receive_predictor[512kb->512kb]
>> [2014-04-02 15:19:23,218][DEBUG][org.elasticsearch.client.transport] [
>> Humus Sapien] node_sampler_interval[5s]
>> [2014-04-02 15:19:23,232][DEBUG][org.elasticsearch.client.transport] [
>> Humus Sapien] adding address [[
>> #transport#-1][d][inet[localhost/127.0.0.1:9300]]]
>> [2014-04-02 15:19:23,232][TRACE][org.elasticsearch.client.transport] [
>> Humus Sapien] connecting to listed node (light) [[
>> #transport#-1][d][inet[localhost/127.0.0.1:9300]]]
>> [2014-04-02 15:19:23,233][TRACE][org.elasticsearch.transport.netty] [Jean 
>> Grey-Summers] channel opened: [id: 0x6045a678, /127.0.0.1:51541 => /127.0
>> .0.1:9300]
>> [2014-04-02 15:19:23,233][DEBUG][org.elasticsearch.transport.netty] [
>> Humus Sapien] connected to node [[
>> #transport#-1][d][inet[localhost/127.0.0.1:9300]]]
>> [2014-04-02 15:19:23,250][TRACE][org.elasticsearch.plugins] [Jean Grey-
>> Summers] starting to fetch info on plugins
>> [2014-04-02 15:19:23,262][WARN ][org.elasticsearch.client.transport] [
>> Humus Sapien] node [#transport#-1][d][inet[localhost/127.0.0.1:9300]] 
>> not part of the cluster Cluster [elasticsearch], ignoring...
>>
>>
>> org.elasticsearch.client.transport.NoNodeAvailableException: No node 
>> available
>>  at org.elasticsearch.client.transport.TransportClientNodesService.
>> execute(TransportClientNodesService.java:219)
>>  at org.elasticsearch.client.transport.support.InternalTransportClient.
>> execute(InternalTransportClient.java:106)
>>  at org.elasticsearch.client.support.AbstractClient.index(AbstractClient.
>> java:82)
>>  at org.elasticsearch.client.transport.TransportClient.index(
>> TransportClient.java:330)
>>  at org.elasticsearch.action.index.IndexRequestBuilder.doExecute(
>> IndexRequestBuilder.java:314)
>>  at org.elasticsearch.action.ActionRequestBuilder.execute(
>> ActionRequestBuilder.java:85)
>>  at org.elasticsearch.action.ActionRequestBuilder.execute(
>> ActionRequestBuilder.java:59)
>>  at com.mycode.estests.test.TransportTest.transportTest(TransportTest.
>> java:20)
>>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
>> java:57)
>>  at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>> DelegatingMethodAccessorImpl.java:43)
>>  at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(
>> FrameworkMethod.java:47)
>>  at org.junit.internal.runners.model.ReflectiveCallable.run(
>> ReflectiveCallable.java:12)
>>  at org.junit.runners.model.FrameworkMethod.invokeExplosively(
>> FrameworkMethod.java:44)
>>  at org.junit.internal.runners.statements.InvokeMethod.> style="color: #000;" class
>> ...
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/7da21dd1-e3cd-4e3e-8636-1eabb736593d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: transport.tcp.port doesn't work for localhost?

[Igor Motov]

If you want to be able to connect to it using Transport Client - yes or 
remove it completely. If you still get some failure - post here complete 
log.

On Wednesday, April 2, 2014 10:09:16 AM UTC-4, Dario Rossi wrote:
>
> So shall I set local to false?
>
> Il giorno mercoledì 2 aprile 2014 15:06:04 UTC+1, Igor Motov ha scritto:
>>
>> You are starting local node, which is using local transport, which is not 
>> listening on port 9300. The log message that you see is from transport 
>> client that tries to connect to port 9300 but cannot. Try starting just 
>> your node and you will be see that nobody listens on port 9300.
>>
>> On Tuesday, April 1, 2014 12:52:41 PM UTC-4, Dario Rossi wrote:
>>>
>>> I've the following problem: to do an integration test, I set up an 
>>> embedded node and then I create a TransportClient to connect to it. 
>>>
>>> The setup of the embedded node is (among other things):
>>>
>>>
>>>  port = 11547; // User ports range 1024 - 49151
>>> tcpport = 9300;
>>> settings.put("http.port", port);
>>> settings.put("transport.tcp.port", tcpport);
>>>  
>>> Settings esSettings = settings.build();
>>>
>>>
>>> node = NodeBuilder.nodeBuilder().local(true).settings(esSettings
>>> ).node();  //I tried setting local to false too
>>> node.start();
>>>
>>>
>>> and the transportclient is as simple as:
>>>
>>>
>>>
>>>
>>>   TransportClient client = new TransportClient();
>>> client.addTransportAddress(new InetSocketTransportAddress(
>>> "localhost", 9300));
>>>
>>>
>>> client.prepareIndex("test", "type").setSource("field", "value").
>>> execute().actionGet();
>>>
>>>
>>>
>>>
>>> (I tried both localhost and 127.0.0.1). 
>>>
>>> Anyway I get a connection refused when running the above code:
>>>
>>>
>>> Caused by: java.net.ConnectException: Connection refused: localhost/
>>> 127.0.0.1:9300
>>>  at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>  at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:
>>> 708)
>>>  at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.
>>> connect(NioClientBoss.java:150)
>>>  at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.
>>> processSelectedKeys(NioClientBoss.java:105)
>>>  at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.
>>> process(NioClientBoss.java:79)
>>>  at org.elasticsearch.common.netty.channel.socket.nio.
>>> AbstractNioSelector.run(AbstractNioSelector.java:318)
>>>  at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(
>>> NioClientBoss.java:42)
>>>  at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(
>>> ThreadRenamingRunnable.java:108)
>>>  at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.
>>> run(DeadLockProofWorker.java:42)
>>>  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor
>>> .java:1145)
>>>  at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>> ThreadPoolExecutor.java:615)
>>>  at java.lang.Thread.run(Thread.java:724)
>>> [2014-04-01 17:48:10,836][TRACE][org.elasticsearch.transport.netty] [Cap 'N 
>>> Hawk] connect exception caught on transport layer [[id: 0x9526b405]]
>>> java.net.ConnectException: Connection refused: localhost/127.0.0.1:9300
>>>  at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>  at 
>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
>>>  at 
>>> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
>>>  at 
>>> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
>>>  at 
>>> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
>>>  at 
>>> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
>>>  at 
>>> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
>>>  at 
>>> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)

Re: wait_for_completion doesn't seem to be working when making a snapshot

[Igor Motov]

The wait_for_completion flag has to be specified on URL not in the body. 
Try this:

curl -XPUT "
http://localhost:9200/_snapshot/backup/snapshot_kibana?wait_for_completion=true&pretty"
 
-d '{
"indices": "kibana-int",
"ignore_unavailable": true,
"include_global_state": false
}'

On Wednesday, April 2, 2014 9:43:14 AM UTC-4, Robin Clarke wrote:
>
> I am writing a small script to create a snapshot of my kibana-int index, 
> and hit an odd race condition.
>
> I delete the old snapshot if it exists:
> curl -XDELETE '
> http://localhost:9200/_snapshot/backup/snapshot_kibana?pretty'
>
> Then make the new snapshot
> curl -XPUT "http://localhost:9200/_snapshot/backup/snapshot_kibana?pretty"; 
> -d '{
> "indices": "kibana-int",
> "ignore_unavailable": true,
> "wait_for_completion": true,
> "include_global_state": false
> }'
>
> Then create a tarball of the backup to transfer to another machine
> tar czf /DATA/elasticsearch/kibana-int.tgz -C /DATA/elasticsearch ./backup
>
> When scripted, it seems that the DELETE, and/or PUT are not complete, and 
> I get this error:
> tar: ./backup/indices/kibana-int/4: file changed as we read it
> tar: ./backup/indices/kibana-int/2: file changed as we read it
> tar: ./backup/indices/kibana-int/3: file changed as we read it
>
> I tried putting a sleep between the DELETE and the PUT, and same error, so 
> it seems that perhaps the "wait_for_completion" is not doing what it 
> should...?
>
> Any ideas, other than just putting a sleep in there?
>
> Cheers!
>
> -Robin-
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/beb0d9d0-73ef-4ce1-8e1f-c70e083d65d5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: transport.tcp.port doesn't work for localhost?

[Igor Motov]

You are starting local node, which is using local transport, which is not 
listening on port 9300. The log message that you see is from transport 
client that tries to connect to port 9300 but cannot. Try starting just 
your node and you will be see that nobody listens on port 9300.

On Tuesday, April 1, 2014 12:52:41 PM UTC-4, Dario Rossi wrote:
>
> I've the following problem: to do an integration test, I set up an 
> embedded node and then I create a TransportClient to connect to it. 
>
> The setup of the embedded node is (among other things):
>
>
>  port = 11547; // User ports range 1024 - 49151
> tcpport = 9300;
> settings.put("http.port", port);
> settings.put("transport.tcp.port", tcpport);
>  
> Settings esSettings = settings.build();
>
>
> node = NodeBuilder.nodeBuilder().local(true).settings(esSettings).
> node();  //I tried setting local to false too
> node.start();
>
>
> and the transportclient is as simple as:
>
>
>
>
>   TransportClient client = new TransportClient();
> client.addTransportAddress(new InetSocketTransportAddress(
> "localhost", 9300));
>
>
> client.prepareIndex("test", "type").setSource("field", "value").
> execute().actionGet();
>
>
>
>
> (I tried both localhost and 127.0.0.1). 
>
> Anyway I get a connection refused when running the above code:
>
>
> Caused by: java.net.ConnectException: Connection refused: localhost/127.0.
> 0.1:9300
>  at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>  at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
>  at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.
> connect(NioClientBoss.java:150)
>  at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.
> processSelectedKeys(NioClientBoss.java:105)
>  at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.
> process(NioClientBoss.java:79)
>  at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.
> run(AbstractNioSelector.java:318)
>  at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(
> NioClientBoss.java:42)
>  at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(
> ThreadRenamingRunnable.java:108)
>  at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run
> (DeadLockProofWorker.java:42)
>  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
> java:1145)
>  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
> java:615)
>  at java.lang.Thread.run(Thread.java:724)
> [2014-04-01 17:48:10,836][TRACE][org.elasticsearch.transport.netty] [Cap 'N 
> Hawk] connect exception caught on transport layer [[id: 0x9526b405]]
> java.net.ConnectException: Connection refused: localhost/127.0.0.1:9300
>  at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>  at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
>  at 
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
>  at 
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
>  at 
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
>  at 
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
>  at 
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
>  at 
> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>  at 
> org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  at java.lang.Thread.run(Thread.java:724)
>
>
>
> my colleague was successful when he tried to connect to another host. but 
> he fails with localhost. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d573c5ca-ada7-48b9-acb5-12cbd925bdee%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Getting "connections idle , closing" log on running facet-script

[Igor Motov]

Vineeth,

The script plugin is not using Apache http client. Are you using it in your 
code?

Igor

On Tuesday, March 25, 2014 11:02:48 AM UTC-4, vineeth mohan wrote:
>
>
> Hi ,
>
> While running a script using - 
> https://github.com/imotov/elasticsearch-facet-script
>
> I am seeing following error in debug log. Is there any settings i can tune 
> to increase this timeout or some other workaround to deal with it.
>
> [2014-03-25 
> 09:53:45,608][DEBUG][org.apache.http.impl.conn.tsccm.ConnPoolByRoute] 
> Closing connections idle longer than 60 SECONDS
> [2014-03-25 
> 09:54:45,608][DEBUG][org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager]
>  
> Closing connections idle longer than 60 SECONDS
> [2014-03-25 
> 09:54:45,608][DEBUG][org.apache.http.impl.conn.tsccm.ConnPoolByRoute] 
> Closing connections idle longer than 60 SECONDS
> [2014-03-25 
> 09:55:45,608][DEBUG][org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager]
>  
> Closing connections idle longer than 60 SECONDS
>
> Thanks
> Vineeth
>  

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/a75cd390-9d2c-407c-b758-164c724066aa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: DELETE snapshot request (which was long-running and had not yet completed) is hung

[Igor Motov]

Hi Swaroop, I tried to reproduce it, but in my case snapshot failed 
with AccessDenied almost immediately after I started it. Any other pointers 
besides PUT only permissions?

On Tuesday, March 11, 2014 1:44:00 AM UTC-4, Swaroop wrote:
>
>  Hi Igor,
>  
> It seems that the S3 bucket had "PUT only permissions".
>  
> Regards,
> Swaroop
>  
>  
> 10.03.2014, 17:40, "Igor Motov" >:
>
> That's strange. Wrong S3 permissions should have caused it to failed 
> immediately. Could you provide any more details about the permissions, so I 
> can reproduce it? Meanwhile, restarting the nodes where primary shards of 
> the stuck index are located is the only option that I can think of. 
>  
> We are working on improving the performance of snapshot cancelation (
> https://github.com/elasticsearch/elasticsearch/pull/5244), but it didn't 
> make it to a release yet.
>
> On Monday, March 10, 2014 4:39:24 AM UTC-4, Swaroop wrote:
>
> Hi, 
>
> I had started a snapshot request on a freshly-indexed ES 1.0.1 cluster 
> with cloud plugin installed, but unfortunately the EC2 access keys 
> configured did not have S3 permissions, so ES was in a weird state, so I 
> sent a DELETE snapshot request and it's stuck for more than a couple of 
> hours, any advice on what to do here to cleanup the snapshot request? Logs 
> don't reveal anything relevant. 
>
> Regards, 
> Swaroop 
>
>  
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearc...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/c9ef80ee-5512-44bd-b301-54496a31f4b6%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/c9ef80ee-5512-44bd-b301-54496a31f4b6%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5eb27214-a53e-40a6-8ce4-29a312df0b63%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: DELETE snapshot request (which was long-running and had not yet completed) is hung

[Igor Motov]

That's strange. Wrong S3 permissions should have caused it to failed 
immediately. Could you provide any more details about the permissions, so I 
can reproduce it? Meanwhile, restarting the nodes where primary shards of 
the stuck index are located is the only option that I can think of. 

We are working on improving the performance of snapshot cancelation 
(https://github.com/elasticsearch/elasticsearch/pull/5244), but it didn't 
make it to a release yet.

On Monday, March 10, 2014 4:39:24 AM UTC-4, Swaroop wrote:
>
> Hi, 
>
> I had started a snapshot request on a freshly-indexed ES 1.0.1 cluster 
> with cloud plugin installed, but unfortunately the EC2 access keys 
> configured did not have S3 permissions, so ES was in a weird state, so I 
> sent a DELETE snapshot request and it's stuck for more than a couple of 
> hours, any advice on what to do here to cleanup the snapshot request? Logs 
> don't reveal anything relevant. 
>
> Regards, 
> Swaroop 
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/c9ef80ee-5512-44bd-b301-54496a31f4b6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.