Thank you all very much for all the responses so far.  I've enjoyed reading
them!  We have noticed that storing data inside of Solr results in
significantly worse performance (particularly faceting); so we store the
values of all the fields elsewhere, but index all the data with Solr
Cloud.  I think the suggestion about splitting the data up into blocks of
date/time is where we would be headed.  Having two Solr-Cloud clusters -
one to handle ~30 days of data, and one to handle historical.  Another
option is to use a single Solr Cloud cluster, but use multiple
cores/collections.  Either way you'd need a job to come through and clean
up old data. The historical cluster would have much worse performance,
particularly for clustering and faceting the data, but that may be
acceptable.
I don't know what you mean by 'entity recognition in the queries' - could
you elaborate?

We would want to index and potentially facet on any of the fields - for
example entities_media_url, username, even background color, but we do not
know a-priori what fields will be important to users.
As to why we would want to make the data searchable; well - I don't make
the rules!  Tweets is not the only data source, but it's certainly the
largest that we are currently looking at handling.

I will read up on the Berlin Buzzwords - thank you for the info!

-Joe



On Fri, Mar 4, 2016 at 9:59 AM, Jack Krupansky <jack.krupan...@gmail.com>
wrote:

> As always, the initial question is how you intend to query the data - query
> drives data modeling. How real-time do you need queries to be? How fast do
> you need archive queries to be? How many fields do you need to query on?
> How much entity recognition do you need in queries?
>
>
> -- Jack Krupansky
>
> On Fri, Mar 4, 2016 at 4:19 AM, Charlie Hull <char...@flax.co.uk> wrote:
>
> > On 03/03/2016 19:25, Toke Eskildsen wrote:
> >
> >> Joseph Obernberger <joseph.obernber...@gmail.com> wrote:
> >>
> >>> Hi All - would it be reasonable to index the Twitter 'firehose'
> >>> with Solr Cloud - roughly 500-600 million docs per day indexing
> >>> each of the fields (about 180)?
> >>>
> >>
> >> Possible, yes. Reasonable? It is not going to be cheap.
> >>
> >> Twitter index the tweets themselves and have been quite open about
> >> how they do it. I would suggest looking for their presentations;
> >> slides or recordings. They have presented at Berlin Buzzwords and
> >> Lucene/Solr Revolution and probably elsewhere too. The gist is that
> >> they have done a lot of work and custom coding to handle it.
> >>
> >
> > As I recall they're not using Solr, but rather an in-house layer built on
> > a customised version of Lucene. They're indexing around half a trillion
> > tweets.
> >
> > If the idea is to provide a searchable archive of all tweets, my first
> > question would be 'why': if the idea is to monitor new tweets for
> > particular patterns there are better ways to do this (Luwak for example).
> >
> > Charlie
> >
> >
> >> If I were to guess at a sharded setup to handle such data, and keep
> >>> 2 years worth, I would guess about 2500 shards.  Is that
> >>> reasonable?
> >>>
> >>
> >> I think you need to think well beyond standard SolrCloud setups. Even
> >> if you manage to get 2500 shards running, you will want to do a lot
> >> of tweaking on the way to issue queries so that each request does not
> >> require all 2500 shards to be searched. Prioritizing newer material
> >> and only query the older shards if there is not enough resent results
> >> is an example.
> >>
> >> I highly doubt that a single SolrCloud is the best answer here. Maybe
> >> one cloud for each month and a lot of external logic?
> >>
> >> - Toke Eskildsen
> >>
> >>
> >
> > --
> > Charlie Hull
> > Flax - Open Source Enterprise Search
> >
> > tel/fax: +44 (0)8700 118334
> > mobile:  +44 (0)7767 825828
> > web: www.flax.co.uk
> >
>

Reply via email to