Hi Mark / Ben,
I reopened DS-615 - https://jira.duraspace.org/browse/DS-615
<https://jira.duraspace.org/browse/DS-615>And commented:
Discovery adds another core to solr, search, which I don't believe gets
optimized when the statistics optimize is called. This should be available
as an argument for update-discovery-index -o, and would need to be added to
IndexClient, and SolrServiceImpl.
I recommend adding this performance fix in to 1.7, especially if instances
are going to use discovery.
Do one of you want to commit this, or would you be fine with me coding it
together, its very simple.
The question I have about IndexClient is:
Do we want to call optimize directly from the implementation:
else if(line.hasOption("o")) {
log.info("Optimize Solr search index");
SolrServiceImpl ssi = new SolrServiceImpl();
ssi.optimizeSOLRSearch();
}
Or add a generic maintenance function to the IndexingService interface, and
our implementation in SolrServiceImpl essentially does getSolr().optimize();
else if(line.hasOption("m")) {
log.info("Running maintenance on the Index");
indexer.maintenance();
}
Thanks
--
Peter Dietz
Systems Developer/Engineer
Ohio State University Libraries
On Fri, Oct 15, 2010 at 5:30 PM, Peter Dietz <dietz...@osu.edu> wrote:
> Hi All,
>
> solr.optimize has gotten positive notes from many people so will commit it
> to trunk in time for 1.7. It has no nasty side-effects. If people are
> running cron tasks, then we may want to consider adding that to the
> documentation of best practices.
>
>
> My question for the group is about solr.commit and solrconfig.xml's
> autocommit.
>
> The problem: whenever we have an activity, such as a UsageEvent (user
> downloads an item, number of downloads in statistics for the bitstream
> should increment) we have:
> solr.add(document);
> solr.commit();
>
> Add is a cheap operation, commit is an expensive operation. Too many
> commits per time period will exhaust resources and cause a constant slew of
> errors. (maxWarmingSearchers)
>
> An easy solution would be to remove hardcoded solr.commit, and configure
> solr to use its own inbuilt property of autoCommit. This can be configured
> to automatically commit to solr after a certain amount of time has passed,
> or a certain number of documents are waiting to be committed.
>
>
> So, I'd like to hear input from those who have an opinion on this, in
> particular I imagine that Mark Diggory or someone from Atmire would have
> some useful input.
>
> Thank you all,
>
>
> --
> Peter Dietz
> Systems Developer/Engineer
> Ohio State University Libraries
>
>
>
> 2010/9/14 Claudia Jürgen (JIRA) <a...@dspace.org>
>
>
>> [
>> http://jira.dspace.org/jira/browse/DS-615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel]
>>
>> Claudia Jürgen updated DS-615:
>> ------------------------------
>>
>> Attachment: SolrOptimizeAndAutoCommit.patch
>>
>> Hi Peter,
>>
>> we use solr optimize as a cron job on our 1.6.0 live instance and it works
>> fine.
>>
>> Just one note, you are using autocommit, shouldn't the patch hence enable
>> it via the solrconfig.xml
>> <autoCommit>
>> <maxDocs>10000</maxDocs>
>> <maxTime>1000</maxTime>
>> </autoCommit>
>>
>> patch with solconfig.xml attached.
>>
>> Claudia
>>
>> > Ability to perform maintenance on SOLR with solr.optimize
>> > ---------------------------------------------------------
>> >
>> > Key: DS-615
>> > URL: http://jira.dspace.org/jira/browse/DS-615
>> > Project: DSpace 1.x
>> > Issue Type: Improvement
>> > Components: Solr
>> > Environment: solr
>> > Reporter: Peter Dietz
>> > Assignee: Peter Dietz
>> > Priority: Minor
>> > Fix For: 1.7.0
>> >
>> > Attachments: solr-optimize.patch,
>> SolrOptimizeAndAutoCommit.patch
>> >
>> >
>> > By adding all historical log data to a SOLR index, as well as the
>> constant addition of new records, it might be good for the efficient
>> performance of a SOLR index to periodically run solr.optimize.
>> > Therefore I've created a patch which allows you to run solr.optimize on
>> your solr instance from the command line. This could allow you to add a cron
>> task that runs this periodic maintenance. It also spits out the amount of
>> time taken to run the optimize task.
>> > Once you patch your instance. You can execute this with
>> > /dspace/bin/dspace stats-util --optimize
>> > or
>> > /dspace/bin/dspace stats-util -o
>> > Output looks like:
>> > SOLR Optimize -- Process Started:1277486321673
>> > SOLR Optimize -- Process Finished:1277486321738
>> > SOLR Optimize -- Total time taken:65 (ms).
>> > It would be useful to profile the amount of time taken for solr
>> operations add/update/query before running this optimize task, and then
>> afterwards.
>>
>> --
>> This message is automatically generated by JIRA.
>> -
>> If you think it was sent incorrectly contact one of the administrators:
>> http://jira.dspace.org/jira/secure/Administrators.jspa
>> -
>> For more information on JIRA, see: http://www.atlassian.com/software/jira
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Start uncovering the many advantages of virtual appliances
>> and start using them to simplify application deployment and
>> accelerate your shift to cloud computing.
>> http://p.sf.net/sfu/novell-sfdev2dev
>> _______________________________________________
>> Dspace-devel mailing list
>> Dspace-devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/dspace-devel
>>
>
>
------------------------------------------------------------------------------
Nokia and AT&T present the 2010 Calling All Innovators-North America contest
Create new apps & games for the Nokia N8 for consumers in U.S. and Canada
$10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing
Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store
http://p.sf.net/sfu/nokia-dev2dev
_______________________________________________
Dspace-devel mailing list
Dspace-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-devel
