Hi All,
solr.optimize has gotten positive notes from many people so will commit it
to trunk in time for 1.7. It has no nasty side-effects. If people are
running cron tasks, then we may want to consider adding that to the
documentation of best practices.
My question for the group is about solr.commit and solrconfig.xml's
autocommit.
The problem: whenever we have an activity, such as a UsageEvent (user
downloads an item, number of downloads in statistics for the bitstream
should increment) we have:
solr.add(document);
solr.commit();
Add is a cheap operation, commit is an expensive operation. Too many commits
per time period will exhaust resources and cause a constant slew of errors.
(maxWarmingSearchers)
An easy solution would be to remove hardcoded solr.commit, and configure
solr to use its own inbuilt property of autoCommit. This can be configured
to automatically commit to solr after a certain amount of time has passed,
or a certain number of documents are waiting to be committed.
So, I'd like to hear input from those who have an opinion on this, in
particular I imagine that Mark Diggory or someone from Atmire would have
some useful input.
Thank you all,
--
Peter Dietz
Systems Developer/Engineer
Ohio State University Libraries
2010/9/14 Claudia Jürgen (JIRA) <a...@dspace.org>
>
> [
> http://jira.dspace.org/jira/browse/DS-615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel]
>
> Claudia Jürgen updated DS-615:
> ------------------------------
>
> Attachment: SolrOptimizeAndAutoCommit.patch
>
> Hi Peter,
>
> we use solr optimize as a cron job on our 1.6.0 live instance and it works
> fine.
>
> Just one note, you are using autocommit, shouldn't the patch hence enable
> it via the solrconfig.xml
> <autoCommit>
> <maxDocs>10000</maxDocs>
> <maxTime>1000</maxTime>
> </autoCommit>
>
> patch with solconfig.xml attached.
>
> Claudia
>
> > Ability to perform maintenance on SOLR with solr.optimize
> > ---------------------------------------------------------
> >
> > Key: DS-615
> > URL: http://jira.dspace.org/jira/browse/DS-615
> > Project: DSpace 1.x
> > Issue Type: Improvement
> > Components: Solr
> > Environment: solr
> > Reporter: Peter Dietz
> > Assignee: Peter Dietz
> > Priority: Minor
> > Fix For: 1.7.0
> >
> > Attachments: solr-optimize.patch, SolrOptimizeAndAutoCommit.patch
> >
> >
> > By adding all historical log data to a SOLR index, as well as the
> constant addition of new records, it might be good for the efficient
> performance of a SOLR index to periodically run solr.optimize.
> > Therefore I've created a patch which allows you to run solr.optimize on
> your solr instance from the command line. This could allow you to add a cron
> task that runs this periodic maintenance. It also spits out the amount of
> time taken to run the optimize task.
> > Once you patch your instance. You can execute this with
> > /dspace/bin/dspace stats-util --optimize
> > or
> > /dspace/bin/dspace stats-util -o
> > Output looks like:
> > SOLR Optimize -- Process Started:1277486321673
> > SOLR Optimize -- Process Finished:1277486321738
> > SOLR Optimize -- Total time taken:65 (ms).
> > It would be useful to profile the amount of time taken for solr
> operations add/update/query before running this optimize task, and then
> afterwards.
>
> --
> This message is automatically generated by JIRA.
> -
> If you think it was sent incorrectly contact one of the administrators:
> http://jira.dspace.org/jira/secure/Administrators.jspa
> -
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>
>
>
>
> ------------------------------------------------------------------------------
> Start uncovering the many advantages of virtual appliances
> and start using them to simplify application deployment and
> accelerate your shift to cloud computing.
> http://p.sf.net/sfu/novell-sfdev2dev
> _______________________________________________
> Dspace-devel mailing list
> Dspace-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/dspace-devel
>
------------------------------------------------------------------------------
Download new Adobe(R) Flash(R) Builder(TM) 4
The new Adobe(R) Flex(R) 4 and Flash(R) Builder(TM) 4 (formerly
Flex(R) Builder(TM)) enable the development of rich applications that run
across multiple browsers and platforms. Download your free trials today!
http://p.sf.net/sfu/adobe-dev2dev
_______________________________________________
Dspace-devel mailing list
Dspace-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-devel
