Re: [Dspace-devel] We need to think a bit more about how we use the 'statistics' Solr core

2015-03-26 Thread helix84
For running from cron it's easier to specify a start date and duration rather than end date (which you have to calculate). Although we'd need to make sure that plays well with differing lengths of months, i.e. we should guarantee that if you specify 31 days in a month that has 28, the 3 days that

Re: [Dspace-devel] We need to think a bit more about how we use the 'statistics' Solr core

2015-03-26 Thread Tim Donohue
I wonder how hard the incremental export is to implement? If it's really not that complex overall, then it seems like it'd be a quick win for just doing the Solr Stats backups in general. If I want to ensure my Solr Stats are safe in DSpace 5, my only real option is to back them up via a CSV

Re: [Dspace-devel] We need to think a bit more about how we use the 'statistics' Solr core

2015-03-26 Thread Mark H. Wood
This is a great tool for recovering from a sticky situation of our own making. Thank you! I think, though, that at this point we ought to consider it a one-time repair rather than a routine maintenance tool. Leaving Solr as the primary long-term storage for these data seems unwise, and if we

Re: [Dspace-devel] We need to think a bit more about how we use the 'statistics' Solr core

2015-03-26 Thread Andrea Schweer
Hi again, just a quick update before my weekend starts -- I've updated my pull request with code that does a loss-less reindex and also uses the time field in the export queries. It can't actually do incremental exports yet, and the new reindex functionality has to be run using dsrun for now

Re: [Dspace-devel] Place for developer tools specific to DSpace?

2015-03-26 Thread Tim Donohue
+1 to getting these on Github in some way/shape form. We could just do something similar to what Fedora has done, have two separate orgs in GitHub: * Main Codebase: https://github.com/fcrepo4 * Labs : https://github.com/fcrepo4-labs It seems reasonable to me to create a separate DSpace-labs

Re: [Dspace-devel] We need to think a bit more about how we use the 'statistics' Solr core

2015-03-26 Thread Andrea Schweer
Hi, On 27/03/15 02:31, Tim Donohue wrote: I wonder how hard the incremental export is to implement? If it's really not that complex overall, then it seems like it'd be a quick win for just doing the Solr Stats backups in general. It's easy-ish, I just can't think of a way that isn't at

Re: [Dspace-devel] Place for developer tools specific to DSpace?

2015-03-26 Thread Tim Donohue
All, Here you go: https://github.com/DSpace-Labs Currently, all Committers are owners of this new Labs organization. I put a warning on the organization page that not all projects may be suitable for Production use. So, feel free to add projects at will, or recommend projects to add (any

Re: [Dspace-devel] Place for developer tools specific to DSpace?

2015-03-26 Thread Monika C. Mevenkamp
I like the idea of a labs repo ; hydra does the same main: https://github.com/projecthydra labs: https://github.com/projecthydra-labs I would love to see some XMLUI stuff At the moment I am trying to figure all that out - but a few simple examples to look at would be very helpful Another

Re: [Dspace-devel] We need to think a bit more about how we use the 'statistics' Solr core

2015-03-26 Thread helix84
On Thu, Mar 26, 2015 at 2:31 PM, Tim Donohue tdono...@duraspace.org wrote: As a sidenote: with regards to the Authority index, it seems like the data in that index is possible to *repopulate* from the existing metadata in the database (using ./dspace index-authority). So, it seems like that

Re: [Dspace-devel] We need to think a bit more about how we use the 'statistics' Solr core

2015-03-26 Thread helix84
As discussed in DevMtg yesterday, I summed up the current state and possible development directions of statistics on the wiki: https://wiki.duraspace.org/display/DSPACE/DSpace+statistics+-+current+status+and+future+development Regards, ~~helix84

Re: [Dspace-devel] Place for developer tools specific to DSpace?

2015-03-26 Thread helix84
I also have a little tool that is useful to more people than just me and currently lives only on the wiki - dspace-l10n-check.py We've talked for a long time about a contrib repository. Perhaps we should just do it, create a project for it under DSpace and worry abut the structure later when it

Re: [Dspace-devel] Place for developer tools specific to DSpace?

2015-03-26 Thread helix84
Other miscellanea that could be put in such a contrib repo include: * curation tasks * metadata format transformations (XOAI and non-XOAI) * scripts in other languages than Java * these REST API usage examples: https://github.com/BrunoNZ/dspace-rest-requests Regards, ~~helix84