Blob store, blob size & storage mechanism

2015-10-20 Thread Upayavira
Is there a maximum size to objects in the blob store? How are objects stored? As a stored field? I've got some machine learning models that are 2-4Gb in size, and whilst machine learning models is one of the intended uses of the blob store, putting GB of data in it scares me a little. Is it

Re: Blob store, blob size & storage mechanism

2015-10-20 Thread Shalin Shekhar Mangar
No, the maximum size is limited to 2MB for now. The use-case behind the blob store is to store small jars (custom plugins) and stopwords, synonyms etc (even though those aren't usable right now) so maybe we can relax the limits a little bit. However, it is definitely not meant for GBs of data. On

Re: Blob store, blob size & storage mechanism

2015-10-20 Thread Jack Krupansky
It's unfortunate that the Blob Store API wasn't named "Small File Store API" to convey to users its intended purpose. That said, maybe you could use the same technique as is recommended for large synonym files: Break them into a sequence of smaller files and then take advantage of the fact that

Re: Blob store, blob size & storage mechanism

2015-10-20 Thread Jack Krupansky
I checked the code and the limit is actually 5MB and configurable via the blob.max.size.mb config property. I posted a comment on the Solr doc for this. In any case, thanks for sharing info that you gleaned from the conference, for all of us who couldn't make it. -- Jack Krupansky On Tue, Oct

Re: Blob store, blob size & storage mechanism

2015-10-20 Thread Shalin Shekhar Mangar
Yes, sorry I checked as well and the limit is 5MB. And it is configurable using the property mentioned by Jack. Thanks for correcting me. On Tue, Oct 20, 2015 at 7:48 PM, Jack Krupansky wrote: > I checked the code and the limit is actually 5MB and configurable via > the

Re: Blob store, blob size & storage mechanism

2015-10-20 Thread Upayavira
What is this limit limiting? Is this effectively a stored field, and the bigger it gets, the more issues we'll have with segment merges/etc? Upayavira On Tue, Oct 20, 2015, at 09:25 AM, Shalin Shekhar Mangar wrote: > Yes, sorry I checked as well and the limit is 5MB. And it is > configurable

Re: Blob store, blob size & storage mechanism

2015-10-20 Thread Upayavira
Okay, thx. I heard it mentioned at Lucene Revolution as a location for storing machine learning models. Do people really have models coming in at under 2Mb? It'd be good to get this limitation into the BlobStore docs. Upayavira On Tue, Oct 20, 2015, at 07:19 AM, Shalin Shekhar Mangar wrote: >