Re: Segments marked as deleted?

2015-02-03 Thread David Pilato 
Yeah but if you update a doc, it will delete and insert behind the scene.

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

> Le 3 févr. 2015 à 08:53, Jason Zhang  a écrit :
> 
> Also I forgot to say I didn't do any deletions in this index.
> 
>> On Tuesday, February 3, 2015 at 3:31:02 PM UTC+8, Mark Walkom wrote:
>> ES removes deleted docs via merging, not GC.
>> 
>> You can force a merge using the optimise API - 
>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-optimize.html#indices-optimize
>>  - but be aware that it is resource intensive. Also be sure to read this 
>> blog post - 
>> http://www.elasticsearch.org/blog/lucenes-handling-of-deleted-documents/
>> 
>>> On 3 February 2015 at 18:26, Jason Zhang  wrote:
>>> Hi,
>>> 
>>>   I use segmentspy to detect segment merges in ES. But I find there's 
>>> several "Deleted Docs" which is like "Deleted Docs: 1.55630".
>>>   I know ES will mark items as deleted and remove them later through GC.
>>> 
>>>   But what's the meaning of those deleted segments? Why does ES/lucene not 
>>> remove them? There's no updates in this index since 10 days ago.
>>>   How can I clean them out explicitly?
>>> 
>>>   Thank you.
>>> -- 
>>> You received this message because you are subscribed to the Google Groups 
>>> "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/elasticsearch/8f0458a6-53ee-4e38-a3cc-6564eb693040%40googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/68bacf0e-acff-4cc1-8d79-150a6e40c98b%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/240D0E50-75E6-49E3-B6FB-D41049FBF54D%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Re: Segments marked as deleted?

2015-02-03 Thread joergpra...@gmail.com 
This only means that segment merging is going on. This produces generations
of segments that are obsolete, these are internally marked as "deleted".
Later the obsolete generations are cleaned up automatically when the
segment is touched by significant amount of new data. Do not worry about
that, there is nothing you should do. Of course, you can play with
_optimize API, but the downside is imposing heavy load on the node.

Jörg

On Tue, Feb 3, 2015 at 8:26 AM, Jason Zhang  wrote:

> Hi,
>
>   I use segmentspy to detect segment merges in ES. But I find there's
> several "Deleted Docs" which is like "Deleted Docs: 1.55630".
>   I know ES will mark items as deleted and remove them later through GC.
>
>   But what's the meaning of those deleted segments? Why does ES/lucene not
> remove them? There's no updates in this index since 10 days ago.
>   How can I clean them out explicitly?
>
>   Thank you.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/8f0458a6-53ee-4e38-a3cc-6564eb693040%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHf3pu24P_fEd70i_8kqVjhJQG_27TLuKz99qd08iwhWA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Segments marked as deleted?

2015-02-03 Thread Jason Zhang 
Very little.
Every "Deleted Docs" says like "Deleted Docs: 1.5563", not integer. What 
does the number mean?

The sum may be less than 10 while the index has over 3 million docs.

On Tuesday, February 3, 2015 at 3:52:59 PM UTC+8, Mark Walkom wrote:
>
> How many?
>
> The amount of deleted docs really depends on the datomicity of the index.
>
> On 3 February 2015 at 18:44, Jason Zhang > 
> wrote:
>
>> I've tried `curl -XPOST http://foo:9200/bar/_optimize` 
>> , but there's still some "Deleted Docs" 
>> here. Is this normal?
>>
>> On Tuesday, February 3, 2015 at 3:31:02 PM UTC+8, Mark Walkom wrote:
>>>
>>> ES removes deleted docs via merging, not GC.
>>>
>>> You can force a merge using the optimise API - 
>>> http://www.elasticsearch.org/guide/en/elasticsearch/
>>> reference/current/indices-optimize.html#indices-optimize - but be aware 
>>> that it is resource intensive. Also be sure to read this blog post - 
>>> http://www.elasticsearch.org/blog/lucenes-handling-of-deleted-documents/
>>>
>>> On 3 February 2015 at 18:26, Jason Zhang  wrote:
>>>
 Hi,

   I use segmentspy to detect segment merges in ES. But I find there's 
 several "Deleted Docs" which is like "Deleted Docs: 1.55630".
   I know ES will mark items as deleted and remove them later through GC.

   But what's the meaning of those deleted segments? Why does ES/lucene 
 not remove them? There's no updates in this index since 10 days ago.
   How can I clean them out explicitly?

   Thank you.

 -- 
 You received this message because you are subscribed to the Google 
 Groups "elasticsearch" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to elasticsearc...@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/
 msgid/elasticsearch/8f0458a6-53ee-4e38-a3cc-6564eb693040%
 40googlegroups.com 
 
 .
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/8158bc66-2970-43b3-b771-6bc3d1c6cd87%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f5e0c11d-5f9d-41e7-97c7-2cf943efe3fb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Segments marked as deleted?

2015-02-02 Thread Jason Zhang 
Also I forgot to say I didn't do any deletions in this index.

On Tuesday, February 3, 2015 at 3:31:02 PM UTC+8, Mark Walkom wrote:
>
> ES removes deleted docs via merging, not GC.
>
> You can force a merge using the optimise API - 
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-optimize.html#indices-optimize
>  
> - but be aware that it is resource intensive. Also be sure to read this 
> blog post - 
> http://www.elasticsearch.org/blog/lucenes-handling-of-deleted-documents/
>
> On 3 February 2015 at 18:26, Jason Zhang > 
> wrote:
>
>> Hi,
>>
>>   I use segmentspy to detect segment merges in ES. But I find there's 
>> several "Deleted Docs" which is like "Deleted Docs: 1.55630".
>>   I know ES will mark items as deleted and remove them later through GC.
>>
>>   But what's the meaning of those deleted segments? Why does ES/lucene 
>> not remove them? There's no updates in this index since 10 days ago.
>>   How can I clean them out explicitly?
>>
>>   Thank you.
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/8f0458a6-53ee-4e38-a3cc-6564eb693040%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/68bacf0e-acff-4cc1-8d79-150a6e40c98b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Segments marked as deleted?

2015-02-02 Thread Mark Walkom 
How many?

The amount of deleted docs really depends on the atomicity of the index.

On 3 February 2015 at 18:44, Jason Zhang  wrote:

> I've tried `curl -XPOST http://foo:9200/bar/_optimize`
> , but there's still some "Deleted Docs"
> here. Is this normal?
>
> On Tuesday, February 3, 2015 at 3:31:02 PM UTC+8, Mark Walkom wrote:
>>
>> ES removes deleted docs via merging, not GC.
>>
>> You can force a merge using the optimise API -
>> http://www.elasticsearch.org/guide/en/elasticsearch/
>> reference/current/indices-optimize.html#indices-optimize - but be aware
>> that it is resource intensive. Also be sure to read this blog post -
>> http://www.elasticsearch.org/blog/lucenes-handling-of-deleted-documents/
>>
>> On 3 February 2015 at 18:26, Jason Zhang  wrote:
>>
>>> Hi,
>>>
>>>   I use segmentspy to detect segment merges in ES. But I find there's
>>> several "Deleted Docs" which is like "Deleted Docs: 1.55630".
>>>   I know ES will mark items as deleted and remove them later through GC.
>>>
>>>   But what's the meaning of those deleted segments? Why does ES/lucene
>>> not remove them? There's no updates in this index since 10 days ago.
>>>   How can I clean them out explicitly?
>>>
>>>   Thank you.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/elasticsearch/8f0458a6-53ee-4e38-a3cc-6564eb693040%
>>> 40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/8158bc66-2970-43b3-b771-6bc3d1c6cd87%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8F0FznwO3QBbCkvTwMD2H_FpyRcDdu8xoPLr6JCU%3D8CA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Segments marked as deleted?

2015-02-02 Thread Jason Zhang 
I've tried `curl -XPOST http://foo:9200/bar/_optimize`, but there's still 
some "Deleted Docs" here. Is this normal?

On Tuesday, February 3, 2015 at 3:31:02 PM UTC+8, Mark Walkom wrote:
>
> ES removes deleted docs via merging, not GC.
>
> You can force a merge using the optimise API - 
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-optimize.html#indices-optimize
>  
> - but be aware that it is resource intensive. Also be sure to read this 
> blog post - 
> http://www.elasticsearch.org/blog/lucenes-handling-of-deleted-documents/
>
> On 3 February 2015 at 18:26, Jason Zhang > 
> wrote:
>
>> Hi,
>>
>>   I use segmentspy to detect segment merges in ES. But I find there's 
>> several "Deleted Docs" which is like "Deleted Docs: 1.55630".
>>   I know ES will mark items as deleted and remove them later through GC.
>>
>>   But what's the meaning of those deleted segments? Why does ES/lucene 
>> not remove them? There's no updates in this index since 10 days ago.
>>   How can I clean them out explicitly?
>>
>>   Thank you.
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/8f0458a6-53ee-4e38-a3cc-6564eb693040%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/8158bc66-2970-43b3-b771-6bc3d1c6cd87%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Segments marked as deleted?

2015-02-02 Thread Mark Walkom 
ES removes deleted docs via merging, not GC.

You can force a merge using the optimise API -
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-optimize.html#indices-optimize
- but be aware that it is resource intensive. Also be sure to read this
blog post -
http://www.elasticsearch.org/blog/lucenes-handling-of-deleted-documents/

On 3 February 2015 at 18:26, Jason Zhang  wrote:

> Hi,
>
>   I use segmentspy to detect segment merges in ES. But I find there's
> several "Deleted Docs" which is like "Deleted Docs: 1.55630".
>   I know ES will mark items as deleted and remove them later through GC.
>
>   But what's the meaning of those deleted segments? Why does ES/lucene not
> remove them? There's no updates in this index since 10 days ago.
>   How can I clean them out explicitly?
>
>   Thank you.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/8f0458a6-53ee-4e38-a3cc-6564eb693040%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9nurQur0KfHSv9TJgQf9FsoRSKNXOLUjAJaVbnhp9DEg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Segments marked as deleted?

2015-02-02 Thread Jason Zhang 
Hi,

  I use segmentspy to detect segment merges in ES. But I find there's 
several "Deleted Docs" which is like "Deleted Docs: 1.55630".
  I know ES will mark items as deleted and remove them later through GC.

  But what's the meaning of those deleted segments? Why does ES/lucene not 
remove them? There's no updates in this index since 10 days ago.
  How can I clean them out explicitly?

  Thank you.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/8f0458a6-53ee-4e38-a3cc-6564eb693040%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.