Hi Ondřej, I wonder if you'd be willing to submit a Pull Request of these changes (the new OAIIndexEventConsumer, etc)? These seem like potentially useful changes overall, and would be something we'd consider adding to DSpace.
My only (small) recommendation would be to simply put the OAIIndexEventConsumer into the "dspace-api", so that it can be easily enabled/disabled from the "event.dispatcher.default.consumers" setting. Plus, if it's in the dspace-api, that ensures it'd also be triggered by any updates from other interface (XMLUI, JSPUI, REST, etc) - Tim On 5/27/2015 5:08 AM, Ondřej Košarko wrote: > > ---------- Forwarded message ---------- > From: *Ondrej Kosarko* <[email protected] <mailto:[email protected]>> > Date: 2015-05-27 12:05 GMT+02:00 > Subject: Re: [Dspace-tech] OAI-PMH data provider 2.0 not persistent? > To: Christian Scheible <[email protected] > <mailto:[email protected]>> > Cc: "[email protected] > <mailto:[email protected]>" > <[email protected] > <mailto:[email protected]>> > > > Hi Christian, > > I'm not sure I remember all the setup, but here goes... > > in dspace.cfg there is > > event.dispatcher.default.consumers = versioning, discovery, eperson, > harvester, eudatreplication, *xoai* > > and > > # consumer to maintain the oai index > event.consumer.xoai.class = cz.cuni.mff.ufal.event.OAIIndexEventConsumer > event.consumer.xoai.filters = > Community|Collection|Item|Bundle|Bitstream+Add|Create|Modify|Modify_Metadata|Delete|Remove > > > and you need to make sure the class is found by the event dispatcher, we > solve it by adding dspace-oai(where the consumer lives) as a dependency > to dspace-xmlui > > diff --git a/dspace-xmlui/pom.xml b/dspace-xmlui/pom.xml > index 17d75c0..f19257f 100644 > --- a/dspace-xmlui/pom.xml > +++ b/dspace-xmlui/pom.xml > @@ -93,6 +93,22 @@ > </profiles> > <dependencies> > + <!-- because of event listener that updates oai --> > + <dependency> > + <groupId>org.dspace</groupId> > + <artifactId>dspace-oai</artifactId> > + <classifier>classes</classifier> > + </dependency> > > I think that's about it... > > Regards, > OK > > > 2015-05-27 9:05 GMT+02:00 Christian Scheible > <[email protected] > <mailto:[email protected]>>: > > Hi Ondřej, > > I have seen that you are using your own OAIIndexEventConsumer. We > did the same but I added It in the module additions (which leads to > code duplication). > How do you register your Consumer? Is it the same way as for other > consumers? Because I couldn't solve that problem. > > Best Regards, > Christian > > > Am 27.05.2015 um 08:41 schrieb Ondřej Košarko: >> In short no... >> >> As I've suggested in the thread the indexAll should include also >> the withdrawn, that's what we do >> >> https://github.com/ufal/lindat-dspace/blob/lindat/dspace-oai/src/main/java/org/dspace/xoai/app/XOAI.java#L193 >> We use our Event consumer to trigger the update (it currently >> doesn't handle caching and changes in discoverable flag) >> >> https://github.com/ufal/lindat-dspace/blob/lindat/dspace-oai/src/main/java/cz/cuni/mff/ufal/event/OAIIndexEventConsumer.java >> And by the way dspace changed the deletion mode to transient in >> https://jira.duraspace.org/browse/DS-2491 so maybe the right way >> for dspace is updating the documentation... >> >> >> OK >> >> >> >> 2015-05-26 23:19 GMT+02:00 Jozef Misutka <[email protected] >> <mailto:[email protected]>>: >> >> Ondrej, is this happening to us as well? >> >> jm >> >> >> ------------------------------------------------------------------------ >> *From: *"Tim Donohue" <[email protected] >> <mailto:[email protected]>> >> *To: *[email protected] >> <mailto:[email protected]> >> *Sent: *Tuesday, 26 May, 2015 23:09:20 >> >> *Subject: *Re: [Dspace-tech] OAI-PMH data provider 2.0 not >> persistent? >> >> Hi All, >> >> I'm only just now "re-discovering" this thread, as I've found >> this same >> behavior in DSpace OAI-PMH servers. Here's how to see this >> behavior (and >> I've double checked it on http://demo.dspace.org, running 5.2): >> >> 1. Create an Item >> 2. Run the OAI-PMH import (./dspace oai import). >> 3. The item now appears in OAI-PMH >> 4. Withdraw that Item >> 5. Re-Run the OAI-PMH import (./dspace oai import) >> 6. The Item still appears in OAI-PMH and is NEVER flagged as >> withdrawn. >> >> So, you can still access all its metadata, etc. The ONLY way >> to trigger >> an update to the OAI-PMH record (of the withdrawn item) is to >> re-import >> EVERYTHING (./dspace oai import -c). However, as noted, since >> withdrawn >> items are not included in the XOAI "indexAll()" command, the >> item will >> now disappear from OAI-PMH entirely. >> >> Here's that indexAll() command: >> >> https://github.com/DSpace/DSpace/blob/master/dspace-oai/src/main/java/org/dspace/xoai/app/XOAI.java#L191 >> >> This behavior seems to be counter to what is documented at: >> >> https://wiki.duraspace.org/pages/viewpage.action?pageId=45548245#OAI-PMHDataProvider2.0%28Internals%29-Deletions >> >> The documentation specifically states that "DSpace keeps a >> permanent >> record of withdrawn items". It also states that a request for a >> withdrawn item "will yield the 'record deleted' header". >> >> This sounds like a bug to me. The documentation definitely >> does not >> match with the behavior. >> >> I've created a new bug ticket for this. It will need a >> volunteer to resolve. >> >> https://jira.duraspace.org/browse/DS-2593 >> >> - Tim >> >> On 3/6/2015 5:21 AM, helix84 wrote: >> > Yes, that makes sense. Still, it needs a brief verification >> because >> > sometimes our understanding of a concept and its >> implementation may >> > differ. >> > >> > >> > On Fri, Mar 6, 2015 at 10:12 AM, Kristian Roberto Salcedo >> > <[email protected] <mailto:[email protected]>> wrote: >> >> Hi Ivan, >> >> >> >> I might have overlooked something, but from your comments >> on https://jira.duraspace.org/browse/DS-2491 wouldnt it be >> meaningful >> >> to do both your declaration change and Ondřej's suggestion >> in order to match both what is declared and what the >> documentation says about OAI and deleted items? >> >> >> >> regards, >> >> Kristian >> >> >> >>> -----Original Message----- >> >>> From: [email protected] <mailto:[email protected]> >> [mailto:[email protected] <mailto:[email protected]>] On >> Behalf Of >> >>> helix84 >> >>> Sent: Thursday, March 05, 2015 3:46 PM >> >>> To: Kristian Roberto Salcedo >> >>> Cc: [email protected] >> <mailto:[email protected]>; João Melo >> >>> Subject: Re: [Dspace-tech] OAI-PMH data provider 2.0 not >> persistent? >> >>> >> >>> Hi Kristian, >> >>> >> >>> I think you're right that the declared >> >>> <deletedRecord>persistent</deletedRecord> doesn't match >> how DSpace >> >>> behaves. I filed a Jira issue and created a pull request >> to change the declared >> >>> status to <deletedRecord>transient</deletedRecord>. >> >>> >> >>> https://jira.duraspace.org/browse/DS-2491 >> >>> >> >>> >> >>> Regards, >> >>> ~~helix84 >> >>> >> >>> Compulsory reading: DSpace Mailing List Etiquette >> >>> >> https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette >> >>> >> >>> >> >>> On Thu, Mar 5, 2015 at 2:52 PM, Kristian Roberto Salcedo >> >>> <[email protected] <mailto:[email protected]>> wrote: >> >>>> Hi all, >> >>>> >> >>>> In Dspace 4.2 we are currently seeing that our OAI feeds >> are not being >> >>> persistent as per these definitions when it comes to >> deleted (withdrawn) >> >>> items: >> >>>> >> >>>> >> http://www.openarchives.org/OAI/openarchivesprotocol.html#deletion >> >>>> >> https://wiki.duraspace.org/pages/viewpage.action?pageId=34640887#OAI- >> >>> P >> >>>> MHDataProvider2.0(Internals)-Deletions >> >>>> >> >>>> Withdrawn items do not get the <header status=deleted> as >> they should, >> >>> like in this example: >> >>>> >> >>>> >> >>> >> >> http://webservices.itcs.umich.edu/mediawiki/oaibp/index.php/Deleted_Re >> >>>> cord_Example_1 >> >>>> >> >>>> Withdrawing an item is not reflected in the OAI entry in >> any way. >> >>>> >> >>>> ---------------------------------------------- >> >>>> >> >>>> This is our Identify page at the moment: >> >>>> >> >>>> https://www.duo.uio.no/oai/request?verb=Identify >> >>>> >> >>>> ---------------------------------------------- >> >>>> >> >>>> I believe we're doing everything right regarding config >> parameters and >> >>> maintenance of the solr oai index. >> >>>> >> >>>> The only place I can find a "persistent"-parameter is in >> this config file: >> >>>> /www/var/data/dspace/config/oaicat.properties >> >>>> which I thought was deprecated, but we still set it just >> to be sure: >> >>>> Identify.deletedRecord=persistent >> >>>> >> >>>> All other relevant config parameters are set in these two >> files as far as I can >> >>> tell: >> >>>> >> >>>> /www/var/data/dspace/config/modules/oai.cfg >> >>>> /www/var/data/dspace/config/crosswalks/oai/description.xml >> >>>> >> >>>> We are running the oai import -o command nightly. >> >>>> >> >>>> ---------------------------------------------- >> >>>> >> >>>> The only way we've found to remove withdrawn items from >> the OAI feed is >> >>> by running a complete re-indexing of the oai solr index: >> >>>> >> >>>> /www/var/data/dspace/bin/dspace oai import -c >> >>>> >> >>>> with a subsequent >> >>>> >> >>>> /www/var/data/dspace/bin/dspace oai clean-cache >> >>>> >> >>>> This removes a withdrawn item: >> >>>> >> >>>> https://www.duo.uio.no/handle/10852/42670 >> >>>> >> >>>> completely from the feed: >> >>>> >> >>>> >> >>> >> https://www.duo.uio.no/oai/request?verb=GetRecord&metadataPrefix=oai >> >>> _d >> >>>> c&identifier=oai:localhost:10852/42670 >> >>>> >> >>>> which is not what we want... >> >>>> >> >>>> ---------------------------------------------- >> >>>> >> >>>> >> >>>> Is anyone else experiencing the same problem? >> >>>> >> >>>> Maybe I've missed something - If this actually works >> differently than we >> >>> expect or has been addressed in OAI 2.1, please let me know. >> >>>> >> >>>> >> >>>> regards, >> >>>> Kristian Salcedo >> >>>> Universitetet of Oslo Library >> >>>> Department of digital services >> >>>> >> >>>> >> >>>> >> >> ---------------------------------------------------------------------- >> >>>> -------- Dive into the World of Parallel Programming The >> Go Parallel >> >>>> Website, sponsored by Intel and developed in partnership >> with Slashdot >> >>>> Media, is your hub for all things parallel software >> development, from >> >>>> weekly thought leadership blogs to news, videos, case >> studies, >> >>>> tutorials and more. Take a look and join the conversation >> now. >> >>>> http://goparallel.sourceforge.net/ >> >>>> _______________________________________________ >> >>>> DSpace-tech mailing list >> >>>> [email protected] >> <mailto:[email protected]> >> >>>> https://lists.sourceforge.net/lists/listinfo/dspace-tech >> >>>> List Etiquette: >> >>>> >> https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette >> > >> > >> >> ------------------------------------------------------------------------------ >> > Dive into the World of Parallel Programming The Go Parallel >> Website, sponsored >> > by Intel and developed in partnership with Slashdot Media, >> is your hub for all >> > things parallel software development, from weekly thought >> leadership blogs to >> > news, videos, case studies, tutorials and more. Take a look >> and join the >> > conversation now. http://goparallel.sourceforge.net/ >> > _______________________________________________ >> > DSpace-tech mailing list >> > [email protected] >> <mailto:[email protected]> >> > https://lists.sourceforge.net/lists/listinfo/dspace-tech >> > List Etiquette: >> https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette >> > >> >> >> ------------------------------------------------------------------------------ >> _______________________________________________ >> DSpace-tech mailing list >> [email protected] >> <mailto:[email protected]> >> https://lists.sourceforge.net/lists/listinfo/dspace-tech >> List Etiquette: >> https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette >> >> >> >> >> >> ------------------------------------------------------------------------------ >> >> >> _______________________________________________ >> DSpace-tech mailing list >> [email protected] >> <mailto:[email protected]> >> https://lists.sourceforge.net/lists/listinfo/dspace-tech >> List >> Etiquette:https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette > > > -- > Christian Scheible > Softwareentwickler / Abt. Content-basierte Dienste > Kommunikations-, Informations- und Medienzentrum (KIM) > Universität Konstanz > 78457 Konstanz > +49 (0)7531 / 88-2857 > Raum B 703 > > > > ------------------------------------------------------------------------------ > > _______________________________________________ > DSpace-tech mailing list > [email protected] > <mailto:[email protected]> > https://lists.sourceforge.net/lists/listinfo/dspace-tech > List Etiquette: > https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette > > > > > > ------------------------------------------------------------------------------ > > > > _______________________________________________ > DSpace-tech mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/dspace-tech > List Etiquette: > https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette > ------------------------------------------------------------------------------ _______________________________________________ DSpace-tech mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
