Re: ElasticSearch- IndexReaders cannot exceed 2147483647

[Prasanth R]

Dear All,
Thanks for your replies.

 Conclusion is, we can not store more than 2147483647 records per shard as 
of now. The only option is we need to increase the shard count.

Thanks
Prasath Rajan

On Tuesday, October 14, 2014 9:34:33 PM UTC+5:30, Jörg Prante wrote:
>
> You can not store more than 2G docs per shard in "Lucene 4.x" codecs. This 
> is a documented Lucene limit: 
>
> "Similarly, Lucene uses a Java int to refer to document numbers, and the 
> index file format uses an Int32 on-disk to store document numbers. This is 
> a limitation of both the index file format and the current implementation. 
> Eventually these should be replaced with either UInt64 values, or better 
> yet, VInt values which have no limit."
>
>
> https://lucene.apache.org/core/4_9_1/core/org/apache/lucene/codecs/lucene49/package-summary.html#Limitations
>
> Jörg
>
>
> On Tue, Oct 14, 2014 at 5:37 PM, Prasanth R  > wrote:
>
>> Thanks for the reply.
>>
>> My scenario here is, 
>> 1) No nested docs.
>> 2) I don't have any limit per shard..
>>
>> I didn't know about internal limit of ES.
>> On Oct 14, 2014 8:23 PM, "Alexandre Rafalovitch" > > wrote:
>>
>>> On 14 October 2014 10:33, Prasanth R >> > wrote:
>>> > There is no upper limit...
>>>
>>> Well, then you must have an infinitely scalable architecture and a
>>> decision when the content starts getting shared. So, then the question
>>> is what is your individual shard allowed to grow to. Which is "how
>>> many documents - including nested - you are expecting to have in a
>>> single shard".
>>>
>>> Because, ElasticSearch has an internal limit and you just hit it. So,
>>> the question is whether it is intentional, unintentional or a result
>>> of a bug.
>>>
>>> Regards,
>>>Alex.
>>>
>>> Personal: http://www.outerthoughts.com/ and @arafalov
>>> Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
>>> Solr popularizers community: https://www.linkedin.com/groups?gid=6713853
>>>
>>> --
>>> You received this message because you are subscribed to a topic in the 
>>> Google Groups "elasticsearch" group.
>>> To unsubscribe from this topic, visit 
>>> https://groups.google.com/d/topic/elasticsearch/H74KAYmGtoc/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to 
>>> elasticsearc...@googlegroups.com .
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/elasticsearch/CAEFAe-G8GsLZpeO9-b5_RGs%3D-tGBNp39QgeY2rywjRnOQfcfnw%40mail.gmail.com
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/CAJLGCR1LRDnv9Nw71py%3DPFksSqLnXOG6x6-K0kUW1j26%3D6_pYA%40mail.gmail.com
>>  
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d85eb58f-2c9d-4fba-88ca-55b64a74be3a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ElasticSearch- IndexReaders cannot exceed 2147483647

[joergpra...@gmail.com]

You can not store more than 2G docs per shard in "Lucene 4.x" codecs. This
is a documented Lucene limit:

"Similarly, Lucene uses a Java int to refer to document numbers, and the
index file format uses an Int32 on-disk to store document numbers. This is
a limitation of both the index file format and the current implementation.
Eventually these should be replaced with either UInt64 values, or better
yet, VInt values which have no limit."

https://lucene.apache.org/core/4_9_1/core/org/apache/lucene/codecs/lucene49/package-summary.html#Limitations

Jörg


On Tue, Oct 14, 2014 at 5:37 PM, Prasanth R 
wrote:

> Thanks for the reply.
>
> My scenario here is,
> 1) No nested docs.
> 2) I don't have any limit per shard..
>
> I didn't know about internal limit of ES.
> On Oct 14, 2014 8:23 PM, "Alexandre Rafalovitch" 
> wrote:
>
>> On 14 October 2014 10:33, Prasanth R  wrote:
>> > There is no upper limit...
>>
>> Well, then you must have an infinitely scalable architecture and a
>> decision when the content starts getting shared. So, then the question
>> is what is your individual shard allowed to grow to. Which is "how
>> many documents - including nested - you are expecting to have in a
>> single shard".
>>
>> Because, ElasticSearch has an internal limit and you just hit it. So,
>> the question is whether it is intentional, unintentional or a result
>> of a bug.
>>
>> Regards,
>>Alex.
>>
>> Personal: http://www.outerthoughts.com/ and @arafalov
>> Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
>> Solr popularizers community: https://www.linkedin.com/groups?gid=6713853
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "elasticsearch" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/elasticsearch/H74KAYmGtoc/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CAEFAe-G8GsLZpeO9-b5_RGs%3D-tGBNp39QgeY2rywjRnOQfcfnw%40mail.gmail.com
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAJLGCR1LRDnv9Nw71py%3DPFksSqLnXOG6x6-K0kUW1j26%3D6_pYA%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHFeV%3D6C%2BZWSs85yhZnzU%3D-Eycw4aPZUy4%2BDLYEL_0bXQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: ElasticSearch- IndexReaders cannot exceed 2147483647

[Prasanth R]

Thanks for the reply.

My scenario here is,
1) No nested docs.
2) I don't have any limit per shard..

I didn't know about internal limit of ES.
On Oct 14, 2014 8:23 PM, "Alexandre Rafalovitch"  wrote:

> On 14 October 2014 10:33, Prasanth R  wrote:
> > There is no upper limit...
>
> Well, then you must have an infinitely scalable architecture and a
> decision when the content starts getting shared. So, then the question
> is what is your individual shard allowed to grow to. Which is "how
> many documents - including nested - you are expecting to have in a
> single shard".
>
> Because, ElasticSearch has an internal limit and you just hit it. So,
> the question is whether it is intentional, unintentional or a result
> of a bug.
>
> Regards,
>Alex.
>
> Personal: http://www.outerthoughts.com/ and @arafalov
> Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
> Solr popularizers community: https://www.linkedin.com/groups?gid=6713853
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/H74KAYmGtoc/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAEFAe-G8GsLZpeO9-b5_RGs%3D-tGBNp39QgeY2rywjRnOQfcfnw%40mail.gmail.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAJLGCR1LRDnv9Nw71py%3DPFksSqLnXOG6x6-K0kUW1j26%3D6_pYA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: ElasticSearch- IndexReaders cannot exceed 2147483647

[Alexandre Rafalovitch]

On 14 October 2014 10:33, Prasanth R  wrote:
> There is no upper limit...

Well, then you must have an infinitely scalable architecture and a
decision when the content starts getting shared. So, then the question
is what is your individual shard allowed to grow to. Which is "how
many documents - including nested - you are expecting to have in a
single shard".

Because, ElasticSearch has an internal limit and you just hit it. So,
the question is whether it is intentional, unintentional or a result
of a bug.

Regards,
   Alex.

Personal: http://www.outerthoughts.com/ and @arafalov
Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
Solr popularizers community: https://www.linkedin.com/groups?gid=6713853

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEFAe-G8GsLZpeO9-b5_RGs%3D-tGBNp39QgeY2rywjRnOQfcfnw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: ElasticSearch- IndexReaders cannot exceed 2147483647

[Alexandre Rafalovitch]

Obvious question: how many documents are you expecting to have in
there? Including the nested ones if you are storing them as separate
documents.

Regards,
   Alex.
Personal: http://www.outerthoughts.com/ and @arafalov
Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
Solr popularizers community: https://www.linkedin.com/groups?gid=6713853


On 14 October 2014 10:25, Prasanth R  wrote:
> Getting following exception in one of our ElasticSearch Index,
>
> [Failed to start shard, message [IndexShardGatewayRecoveryException[[fw][4]
> failed recovery]; nested: IllegalArgumentException[Too many documents,
> composite IndexReaders cannot exceed 2147483647]; ]]
>
> Please somebody help.
>
> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/f60845f2-476d-489a-936f-291884986dbf%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEFAe-EfMGvubuVGK%2Bp0BZpAopj7UOVL%2BUE6c18JDAJc_Am-bw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: ElasticSearch- IndexReaders cannot exceed 2147483647

[Prasanth R]

There is no upper limit... 


On Tuesday, October 14, 2014 8:02:03 PM UTC+5:30, Alexandre Rafalovitch 
wrote:
>
> Obvious question: how many documents are you expecting to have in 
> there? Including the nested ones if you are storing them as separate 
> documents. 
>
> Regards, 
>Alex. 
> Personal: http://www.outerthoughts.com/ and @arafalov 
> Solr resources and newsletter: http://www.solr-start.com/ and @solrstart 
> Solr popularizers community: https://www.linkedin.com/groups?gid=6713853 
>
>
> On 14 October 2014 10:25, Prasanth R > 
> wrote: 
> > Getting following exception in one of our ElasticSearch Index, 
> > 
> > [Failed to start shard, message 
> [IndexShardGatewayRecoveryException[[fw][4] 
> > failed recovery]; nested: IllegalArgumentException[Too many documents, 
> > composite IndexReaders cannot exceed 2147483647]; ]] 
> > 
> > Please somebody help. 
> > 
> > Thanks 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups 
> > "elasticsearch" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an 
> > email to elasticsearc...@googlegroups.com . 
> > To view this discussion on the web visit 
> > 
> https://groups.google.com/d/msgid/elasticsearch/f60845f2-476d-489a-936f-291884986dbf%40googlegroups.com.
>  
>
> > For more options, visit https://groups.google.com/d/optout. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/3646a7e4-7246-43df-8970-01b776a0720f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

ElasticSearch- IndexReaders cannot exceed 2147483647

[Prasanth R]

Getting following exception in one of our ElasticSearch Index,

[Failed to start shard, message [IndexShardGatewayRecoveryException[[fw][4] 
failed recovery]; nested: IllegalArgumentException[Too many documents, 
composite IndexReaders cannot exceed 2147483647]; ]]

Please somebody help.

Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f60845f2-476d-489a-936f-291884986dbf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.