Hi Mark,
Appending session actions just to be able to return more than one
session without retrieving large number of results is not good tradeoff.
Like Upayavira suggested, you should consider storing one action per doc
and aggregate on read time or push to Solr once session ends and
aggregate on some other layer.
If you are thinking handling infrastructure might be too much, you may
consider using some of logging services to hold data. One such service
is Sematext's Logsene (http://sematext.com/logsene).
Thanks,
Emir
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
On 10.02.2016 03:22, Mark Robinson wrote:
Thanks for your replies and suggestions!
Why I store all events related to a session under one doc?
Each session can have about 500 total entries (events) corresponding to it.
So when I try to retrieve a session's info it can back with around 500
records. If it is this compounded one doc per session, I can retrieve more
sessions at a time with one doc per session.
eg under a sessionId an array of eventA activities, eventB activities
(using json). When an eventA activity again occurs, we will read all that
data for that session, append this extra info to evenA data and push the
whole session related data back (indexing) to Solr. Like this for many
sessions parallely.
Why NRT?
Parallely many sessions are being written (4Million sessions hence 4Million
docs per day). A person can do this querying any time.
It is just a look up?
Yes. We just need to retrieve all info for a session and pass it on to
another system. We may even do some extra querying on some data like
timestamps, pageurl etc in that info added to a session.
Thinking of having the data separate from the actual Solr Instance and
mention the loc of the dataDir in solrconfig.
If Solr is not a good option could you please suggest something which will
satisfy this use case with min response time while querying.
Thanks!
Mark
On Tue, Feb 9, 2016 at 6:02 PM, Daniel Collins <danwcoll...@gmail.com>
wrote:
So as I understand your use case, its effectively logging actions within a
user session, why do you have to do the update in NRT? Why not just log
all the user session events (with some unique key, and ensuring the session
Id is in the document somewhere), then when you want to do the query, you
join on the session id, and that gives you all the data records for that
session. I don't really follow why it has to be 1 document (which you
continually update). If you really need that aggregation, couldn't that
happen offline?
I guess your 1 saving grace is that you query using the unique ID (in your
scenario) so you could use the real-time get handler, since you aren't
doing a complex query (strictly its not a search, its a raw key lookup).
But I would still question your use case, if you go the Solr route for that
kind of scale with querying and indexing that much, you're going to have to
throw a lot of hardware at it, as Jack says probably in the order of
hundreds of machines...
On 9 February 2016 at 19:00, Upayavira <u...@odoko.co.uk> wrote:
Bear in mind that Lucene is optimised towards high read lower write.
That is, it puts in a lot of effort at write time to make reading
efficient. It sounds like you are going to be doing far more writing
than reading, and I wonder whether you are necessarily choosing the
right tool for the job.
How would you later use this data, and what advantage is there to
storing it in Solr?
Upayavira
On Tue, Feb 9, 2016, at 03:40 PM, Mark Robinson wrote:
Hi,
Thanks for all your suggestions. I took some time to get the details to
be
more accurate. Please find what I have gathered:-
My data being indexed is something like this.
I am basically capturing all data related to a user session.
Inside a session I have categorized my actions like actionA, actionB
etc..,
per page.
So each time an action pertaining to say actionA or actionB etc.. (in
each
page) happens, it is updated in Solr under that session (sessionId).
So in short there is only one doc pertaining to a single session
(identified by sessionid) in my Solr index and that is retrieved and
updated
whenever a new action under that session occurs.
We expect upto 4Million session per day.
On an average *one session's* *doc has a size* of *3MB to 20MB*.
So if it is *4Million sessions per day*, each session writing around
*500
times to Solr*, it is* 2Billion writes or (indexing) per day to Solr*.
As it is one doc per session, it is *4Million docs per day*.
This is around *80K docs indexed per second* during *peak* hours and
around *15K
docs indexed per second* into Solr during* non-peak* hours.
Number of queries per second is around *320 queries per second*.
1. Average size of a doc
3MB to 20MB
2. Query types:-
Until that session is in progress, whatever data is there for that
session so far is queried and the new action's details captured and
appended to existing data already captured related to that
session
and indexed back into Solr. So, longer the session the more data
retrieved
for each subsequent query to get current data captured for that
session.
Also querying can be done on timestamp etc... which is captured
along
with each action.
3. Are docs grouped somehow?
All data related to a session are retrieved from Solr, updated and
indexed back to Solr based on sessionId. No other grouping.
4. Are they time sensitive (NRT or offline process does this)
As mentioned above this is in NRT. Each time a new user action in
that
session happens, we need to query existing session info already
captured
related to that session and append this new data to this
existing
info retrieved and index it back to Solr.
5. Will they update or it is rebuild every time, etc.
Each time a new user action occurs, the full data pertaining to
that
session so far captured is retrieved from Solr, the extra latest data
pertaining to this new action is appended and indexed back to
Solr.
6. And the other thing you haven't told us is whether you plan on
_adding_
2B docs a day or whether that number is the total corpus size and you
are
re-indexing the 2B docs/day. IOW, if you are adding 2B docs/day, 30
days
later do you have 2B docs or 60B docs in your
corpus?
We are expecting around 4 million sessions per day (per session 500
writes to Solr), which turns out to be 2B indexing done per day. So
after
30 days it would be 4Milion*30 docs in the index.
7. Is there any aging of docs
No we always query against the whole corpus present.
8. Is any doc deleted?
No all data remains in the index
Any suggestion is very welcome!
Thanks!
Mark.
On Mon, Feb 8, 2016 at 3:30 PM, Jack Krupansky <
jack.krupan...@gmail.com
wrote:
Oops... at 100 qps for a single node you would need 120 nodes to get
to 12K
qps and 800 nodes to get 80K qps, but that is just an extremely rough
ballpark estimate, not some precise and firm number. And that's if
all
the
queries can be evenly distributed throughout the cluster and don't
require
fanout to other shards, which effectively turns each incoming query
into n
queries where n is the number of shards.
-- Jack Krupansky
On Mon, Feb 8, 2016 at 12:07 PM, Jack Krupansky <
jack.krupan...@gmail.com>
wrote:
So is there any aging or TTL (in database terminology) of older
docs?
And do all of your queries need to query all of the older documents
all
of
the time or is there a clear hierarchy of querying for aged
documents,
like
past 24-hours vs. past week vs. past year vs. older than a year?
Sure,
you
can always use a function query to boost by the inverse of document
age,
but Solr would be more efficient with filter queries or separate
indexes
for different time scales.
Are documents ever updated or are they write-once?
Are documents explicitly deleted?
Technically you probably could meet those specs, but... how many
organizations have the resources and the energy to do so?
As a back of the envelope calculation, if Solr gave you 100 queries
per
second per node, that would mean you would need 1,200 nodes. It
would
also
depend on whether those queries are very narrow so that a single
node can
execute them or if they require fanout to other shards and then
aggregation
of results from those other shards.
-- Jack Krupansky
On Mon, Feb 8, 2016 at 11:24 AM, Erick Erickson <
erickerick...@gmail.com
wrote:
Short form: You really have to prototype. Here's the long form:
https://lucidworks.com/blog/2012/07/23/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/
I've seen between 20M and 200M docs fit on a single piece of
hardware,
so you'll absolutely have to shard.
And the other thing you haven't told us is whether you plan on
_adding_ 2B docs a day or whether that number is the total corpus
size
and you are re-indexing the 2B docs/day. IOW, if you are adding 2B
docs/day, 30 days later do you have 2B docs or 60B docs in your
corpus?
Best,
Erick
On Mon, Feb 8, 2016 at 8:09 AM, Susheel Kumar <
susheel2...@gmail.com>
wrote:
Also if you are expecting indexing of 2 billion docs as NRT or
if
it
will
be offline (during off hours etc). For more accurate sizing you
may
also
want to index say 10 million documents which may give you idea
how
much
is
your index size and then use that for extrapolation to come up
with
memory
requirements.
Thanks,
Susheel
On Mon, Feb 8, 2016 at 11:00 AM, Emir Arnautovic <
emir.arnauto...@sematext.com> wrote:
Hi Mark,
Can you give us bit more details: size of docs, query types,
are
docs
grouped somehow, are they time sensitive, will they update or
it
is
rebuild
every time, etc.
Thanks,
Emir
On 08.02.2016 16:56, Mark Robinson wrote:
Hi,
We have a requirement where we would need to index around 2
Billion
docs
in
a day.
The queries against this indexed data set can be around 80K
queries
per
second during peak time and during non peak hours around 12K
queries
per
second.
Can Solr realize this huge volumes.
If so, assuming we have no constraints for budget what would
be
a
recommended Solr set up (number of shards, number of Solr
instances
etc...)
Thanks!
Mark
--
Monitoring * Alerting * Anomaly Detection * Centralized Log
Management
Solr & Elasticsearch Support * http://sematext.com/
