Re: [Dspace-devel] We need to think a bit more about how we use the 'statistics' Solr core

2015-03-25 Thread helix84
Hi Tim, CSV export may be adequate for backup, but one important thing suggested here was an event consumer that would write to a persistent store (which could be CSV files). We currently don't have a persistent store. Regards, ~~helix84 Compulsory reading: DSpace Mailing List Etiquette

Re: [Dspace-devel] We need to think a bit more about how we use the 'statistics' Solr core

2015-03-25 Thread Mark H. Wood
On Tue, Mar 24, 2015 at 11:34:44AM -0400, Peter Dietz wrote: Also. What are people thinking would be a safe preservation location for usage events? i.e. for people concerned about resources. What I've been thinking is duplicated DVD-ROMs in fire-insulated storages, right alongside of content

Re: [Dspace-devel] We need to think a bit more about how we use the 'statistics' Solr core

2015-03-25 Thread Tim Donohue
Hi All, Just to bring this thread back to the original question of how we use Solr to store statistics (and also authority info for that matter). Personally, I agree that having statistics authority information stored *solely* in Solr is dangerous. As mentioned, Solr is primarily meant as an

Re: [Dspace-devel] We need to think a bit more about how we use the 'statistics' Solr core

2015-03-25 Thread Monika C. Mevenkamp
I second many of helix’s points Storing Stats data in an external product relies on that product to be around - something that google does not guarantee Whether to use google analytics for anything other than getting nice stats right now very much depends on whether google provides a data

Re: [Dspace-devel] We need to think a bit more about how we use the 'statistics' Solr core

2015-03-25 Thread helix84
On Tue, Mar 24, 2015 at 5:27 PM, TAYLOR Robin robin.tay...@ed.ac.uk wrote: Hi Peter, The short answer is I don't know, but a quick bit of investigation suggests possibly maybe :) . There does appear to be an import facility https://support.google.com/analytics/answer/3191589?hl=en-GB ,

Re: [Dspace-devel] We need to think a bit more about how we use the 'statistics' Solr core

2015-03-25 Thread Tim Donohue
Hi helix84, So, it seems like there's two possible routes to take here: 1. An event consumer writes directly to Solr. The persistent store is then simply a dump from Solr to CSV. 2. An event consumer writes directly to CSV. Solr then indexes those CSVs. So, my question is whether #2 is really

Re: [Dspace-devel] We need to think a bit more about how we use the 'statistics' Solr core

2015-03-25 Thread helix84
Hi Tim, sorry, I think you missed my point about the persistent store. Your 1) is not a persistent store, it's a snapshot of a cache. Yes, I know we've been treating it as if it were a permanent store, but that's what this whole issue is about. It's really not as important what the form or

Re: [Dspace-devel] We need to think a bit more about how we use the 'statistics' Solr core

2015-03-25 Thread Mark H. Wood
On Wed, Mar 25, 2015 at 09:54:02AM -0500, Tim Donohue wrote: Hi helix84, So, it seems like there's two possible routes to take here: 1. An event consumer writes directly to Solr. The persistent store is then simply a dump from Solr to CSV. 2. An event consumer writes directly to CSV.

Re: [Dspace-devel] We need to think a bit more about how we use the 'statistics' Solr core

2015-03-25 Thread Andrea Schweer
Hi all, On 26/03/15 03:07, Tim Donohue wrote: In DSpace 5, we obviously already have a basic version of a backup to CSV for statistics:

Re: [Dspace-devel] We need to think a bit more about how we use the 'statistics' Solr core

2015-03-25 Thread Andrea Schweer
Hi again, On 26/03/15 11:19, Andrea Schweer wrote: Another gap in my code is incremental exports. At the moment, the export part of my code dumps all of the data. I think it would be nice for back-up purposes to be able to specify a start date from which to export, so that people can