Re: Blob store, blob size & storage mechanism
What is this limit limiting? Is this effectively a stored field, and the bigger it gets, the more issues we'll have with segment merges/etc? Upayavira On Tue, Oct 20, 2015, at 09:25 AM, Shalin Shekhar Mangar wrote: > Yes, sorry I checked as well and the limit is 5MB. And it is > configurable using the property mentioned by Jack. Thanks for > correcting me. > > On Tue, Oct 20, 2015 at 7:48 PM, Jack Krupansky > wrote: > > I checked the code and the limit is actually 5MB and configurable via > > the blob.max.size.mb config property. I posted a comment on the Solr doc > > for this. > > > > In any case, thanks for sharing info that you gleaned from the conference, > > for all of us who couldn't make it. > > > > -- Jack Krupansky > > > > On Tue, Oct 20, 2015 at 9:00 AM, Upayavira wrote: > > > >> Okay, thx. I heard it mentioned at Lucene Revolution as a location for > >> storing machine learning models. Do people really have models coming in > >> at under 2Mb? > >> > >> It'd be good to get this limitation into the BlobStore docs. > >> > >> Upayavira > >> > >> On Tue, Oct 20, 2015, at 07:19 AM, Shalin Shekhar Mangar wrote: > >> > No, the maximum size is limited to 2MB for now. The use-case behind > >> > the blob store is to store small jars (custom plugins) and stopwords, > >> > synonyms etc (even though those aren't usable right now) so maybe we > >> > can relax the limits a little bit. However, it is definitely not meant > >> > for GBs of data. > >> > > >> > On Tue, Oct 20, 2015 at 5:26 PM, Upayavira wrote: > >> > > Is there a maximum size to objects in the blob store? How are objects > >> > > stored? As a stored field? > >> > > > >> > > I've got some machine learning models that are 2-4Gb in size, and > >> whilst > >> > > machine learning models is one of the intended uses of the blob store, > >> > > putting GB of data in it scares me a little. Is it reasonable and does > >> > > it work? > >> > > > >> > > Upayavira > >> > > >> > > >> > > >> > -- > >> > Regards, > >> > Shalin Shekhar Mangar. > >> > > > > -- > Regards, > Shalin Shekhar Mangar.
Re: Blob store, blob size & storage mechanism
Yes, sorry I checked as well and the limit is 5MB. And it is configurable using the property mentioned by Jack. Thanks for correcting me. On Tue, Oct 20, 2015 at 7:48 PM, Jack Krupansky wrote: > I checked the code and the limit is actually 5MB and configurable via > the blob.max.size.mb config property. I posted a comment on the Solr doc > for this. > > In any case, thanks for sharing info that you gleaned from the conference, > for all of us who couldn't make it. > > -- Jack Krupansky > > On Tue, Oct 20, 2015 at 9:00 AM, Upayavira wrote: > >> Okay, thx. I heard it mentioned at Lucene Revolution as a location for >> storing machine learning models. Do people really have models coming in >> at under 2Mb? >> >> It'd be good to get this limitation into the BlobStore docs. >> >> Upayavira >> >> On Tue, Oct 20, 2015, at 07:19 AM, Shalin Shekhar Mangar wrote: >> > No, the maximum size is limited to 2MB for now. The use-case behind >> > the blob store is to store small jars (custom plugins) and stopwords, >> > synonyms etc (even though those aren't usable right now) so maybe we >> > can relax the limits a little bit. However, it is definitely not meant >> > for GBs of data. >> > >> > On Tue, Oct 20, 2015 at 5:26 PM, Upayavira wrote: >> > > Is there a maximum size to objects in the blob store? How are objects >> > > stored? As a stored field? >> > > >> > > I've got some machine learning models that are 2-4Gb in size, and >> whilst >> > > machine learning models is one of the intended uses of the blob store, >> > > putting GB of data in it scares me a little. Is it reasonable and does >> > > it work? >> > > >> > > Upayavira >> > >> > >> > >> > -- >> > Regards, >> > Shalin Shekhar Mangar. >> -- Regards, Shalin Shekhar Mangar.
Re: Blob store, blob size & storage mechanism
I checked the code and the limit is actually 5MB and configurable via the blob.max.size.mb config property. I posted a comment on the Solr doc for this. In any case, thanks for sharing info that you gleaned from the conference, for all of us who couldn't make it. -- Jack Krupansky On Tue, Oct 20, 2015 at 9:00 AM, Upayavira wrote: > Okay, thx. I heard it mentioned at Lucene Revolution as a location for > storing machine learning models. Do people really have models coming in > at under 2Mb? > > It'd be good to get this limitation into the BlobStore docs. > > Upayavira > > On Tue, Oct 20, 2015, at 07:19 AM, Shalin Shekhar Mangar wrote: > > No, the maximum size is limited to 2MB for now. The use-case behind > > the blob store is to store small jars (custom plugins) and stopwords, > > synonyms etc (even though those aren't usable right now) so maybe we > > can relax the limits a little bit. However, it is definitely not meant > > for GBs of data. > > > > On Tue, Oct 20, 2015 at 5:26 PM, Upayavira wrote: > > > Is there a maximum size to objects in the blob store? How are objects > > > stored? As a stored field? > > > > > > I've got some machine learning models that are 2-4Gb in size, and > whilst > > > machine learning models is one of the intended uses of the blob store, > > > putting GB of data in it scares me a little. Is it reasonable and does > > > it work? > > > > > > Upayavira > > > > > > > > -- > > Regards, > > Shalin Shekhar Mangar. >
Re: Blob store, blob size & storage mechanism
Okay, thx. I heard it mentioned at Lucene Revolution as a location for storing machine learning models. Do people really have models coming in at under 2Mb? It'd be good to get this limitation into the BlobStore docs. Upayavira On Tue, Oct 20, 2015, at 07:19 AM, Shalin Shekhar Mangar wrote: > No, the maximum size is limited to 2MB for now. The use-case behind > the blob store is to store small jars (custom plugins) and stopwords, > synonyms etc (even though those aren't usable right now) so maybe we > can relax the limits a little bit. However, it is definitely not meant > for GBs of data. > > On Tue, Oct 20, 2015 at 5:26 PM, Upayavira wrote: > > Is there a maximum size to objects in the blob store? How are objects > > stored? As a stored field? > > > > I've got some machine learning models that are 2-4Gb in size, and whilst > > machine learning models is one of the intended uses of the blob store, > > putting GB of data in it scares me a little. Is it reasonable and does > > it work? > > > > Upayavira > > > > -- > Regards, > Shalin Shekhar Mangar.
Re: Blob store, blob size & storage mechanism
It's unfortunate that the Blob Store API wasn't named "Small File Store API" to convey to users its intended purpose. That said, maybe you could use the same technique as is recommended for large synonym files: Break them into a sequence of smaller files and then take advantage of the fact that the synonym file parameter allows a comma-separated list of file names to be specified. Clearly that wouldn't work if you had to specify 2,000 files to get your 4GB in 2MB increments, but maybe you could name the files with a trailing sequence number and then use a wildcard to specify the common prefix for the files. -- Jack Krupansky On Tue, Oct 20, 2015 at 8:19 AM, Shalin Shekhar Mangar < shalinman...@gmail.com> wrote: > No, the maximum size is limited to 2MB for now. The use-case behind > the blob store is to store small jars (custom plugins) and stopwords, > synonyms etc (even though those aren't usable right now) so maybe we > can relax the limits a little bit. However, it is definitely not meant > for GBs of data. > > On Tue, Oct 20, 2015 at 5:26 PM, Upayavira wrote: > > Is there a maximum size to objects in the blob store? How are objects > > stored? As a stored field? > > > > I've got some machine learning models that are 2-4Gb in size, and whilst > > machine learning models is one of the intended uses of the blob store, > > putting GB of data in it scares me a little. Is it reasonable and does > > it work? > > > > Upayavira > > > > -- > Regards, > Shalin Shekhar Mangar. >
Re: Blob store, blob size & storage mechanism
No, the maximum size is limited to 2MB for now. The use-case behind the blob store is to store small jars (custom plugins) and stopwords, synonyms etc (even though those aren't usable right now) so maybe we can relax the limits a little bit. However, it is definitely not meant for GBs of data. On Tue, Oct 20, 2015 at 5:26 PM, Upayavira wrote: > Is there a maximum size to objects in the blob store? How are objects > stored? As a stored field? > > I've got some machine learning models that are 2-4Gb in size, and whilst > machine learning models is one of the intended uses of the blob store, > putting GB of data in it scares me a little. Is it reasonable and does > it work? > > Upayavira -- Regards, Shalin Shekhar Mangar.