Markus, We have had similar issues with the time PROAI takes to process a large number of updates, and have settled for turning off PROAI's polling until we want to run an update. We currently do this automatically by a cron task.
Our timed task enables polling at the appointed hour (currently once per day - late at night) by updating the PROAI table rcAdmin (field pollingEnabled). This will enable polling on PROAI's next fetch cycle (which we have set for 600 seconds) and will start the process. We have a separate task that attempts to turn off polling once every few minutes or so. This task will fail if PROAI is updating because the rcAdmin table is write locked at the start of the process, but will eventually succeed after the process has completed. This makes sure that PROAI doesn't start a batch of updates until the next time we let it. Perhaps this is not the most elegant solution, but it has been trouble free for some time now. The only drawback is that PROAI data is always about a day behind the repository. HTH (and greetings from Ithaca, NY - USA) -Tim Timothy Cornwell, Programmer/Analyst National Science Digital Library (http://nsdl.org) 301 College Avenue Ithaca, NY 14850 > -----Original Message----- > From: Markus Höckner [mailto:[email protected]] > Sent: Wednesday, August 12, 2009 2:27 AM > To: [email protected] > Subject: [Fedora-commons-users] OAI Provider and performance > > > Hi everyone, > > we use the OAI-PMH service of Fedra and have a little > performance problem. > > When we do an batch upload to fedora (up to 600 objects) then > the OAI dies because of the fact that OAI gets the > new/modified objects via risearch (Mulgara) and this takes a while... > So the OAI decides to try it later, but then nothing changed > - it takes to long to get the result from risearch. > A bad sideeffect is, that this endless loop takes 99% CPU -> > so the system is overloaded. > As a consequence we have to kill Tomcat, edit the > lastPollDate in the MySQL DB of proai and restart it. > So the system works but the objects of the batch process > aren't in OAI, because the updated lastPollDate is after the import... > > Is there another way to import the objects from Fedora to PROAI? > What can we do to get more performance for the automatic PROAI import? > Any suggestions? > > Greetings from Austria, > Markus > > > > > > -------------------------------------------------------------- > ---------------- > Let Crystal Reports handle the reporting - Free Crystal > Reports 2008 30-Day > trial. Simplify your report design, integration and > deployment - and focus on > what you do best, core application coding. Discover what's new with > Crystal Reports now. http://p.sf.net/sfu/bobj-july > _______________________________________________ > Fedora-commons-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/fedora-commons-users ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ Fedora-commons-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
