Jesper, I'd say fork master and submit a pull request. By the way, there's a committers call today (Thursday) from 9-9:30am Eastern (UTC-5) if you want to jump in. I think the agenda is mostly around the upcoming release of 3.6.2, but I'd welcome having either (or both) of you and Nicolas join if you're available.
https://wiki.duraspace.org/display/FCREPO/2012-11-08+-+Fedora+Committer+Meeting Eddie On Nov 7, 2012, at 11:12 PM, Jesper Damkjaer <j...@dbc.dk> wrote: > I'll like to take a stab at this one. > I'm not sure though if I can create a branch or if I should create a > fork. > Which do You prefer? > > -Jesper > > On ons, 2012-11-07 at 12:09 +0100, Jesper Damkjaer wrote: >> Yes, I think it would. The problem is that the PID cannot be determined >> until just before the call to objectExists(). And moving the >> pidLock.getWriteLock() up here would essentially do the same as You >> suggest. >> >> -Jesper >> >> On ons, 2012-11-07 at 10:57 +0000, Nicolas Hervé wrote: >>> A lock on a unique object representing the PID to be inserted would be >>> enough to prevent this, no ? >>> ;ppvg usxd9ds=liiiiiiiiii >>> On 06/11/2012 23:51, Jesper Damkjaer wrote: >>>> It might be possible to simply just move the call to >>>> pidLock.getWriteLock() up before the call to objectExists(). >>>> Then the first thread will get the lock and the second thread will be >>>> rejected. Even if the first thread have released the writelock, then it >>>> will have registered the object and the second thread will be rejected >>>> by objectExists(). >>>> >>>> This may of course give a false result if the first thread for some >>>> reason fails to actually create the object. Then both threads are >>>> rejected and no objects are created even though the second thread might >>>> have succeeded (if it had won the race). >>>> I believe we have another kind of locking mechanism which is a >>>> stringLock. This lock doesnt reject the second thread - it just make it >>>> wait until the first thread releases the lock. If this kind of lock is >>>> used, then only one of the threads can enter the last part of >>>> getIngestWriter (assuming the lock is before objectExists()) and if the >>>> first thread succeeds the second thread will, upon release, get an >>>> exception from objectExists(). If on the other hand, the first thread >>>> fails to create an object then, when the lock is released, the second >>>> thread can try to create the object. >>>> >>>> -Jesper >>>> >>>> >>>> On tir, 2012-11-06 at 17:17 -0500, aj...@virginia.edu wrote: >>>>> It's starting to sound like we need a pattern that is less constrictive >>>>> of resources than the use of "synchronized", but still maintains some >>>>> kind of guard against the kind of collision Jesper describes. A >>>>> simpleminded thought: perhaps we could insert a lightweight check against >>>>> the registry before registerObject() with a line of execution that backs >>>>> out any changes if the check fails? Because it's an ingest, those changes >>>>> shouldn't be hard to describe or undo. >>>>> >>>>> --- >>>>> A. Soroka >>>>> Software & Systems Engineering :: Online Library Environment >>>>> the University of Virginia Library >>>>> >>>>> On Nov 6, 2012, at 4:22 PM, Jesper Damkjaer wrote: >>>>> >>>>>> I just had a quick look at the code. As far as I can see (and I may not >>>>>> be right) one problem may be the following: >>>>>> Assume both threads with same PID are inside the method getIngestWriter >>>>>> and has passed the method-call objectExists(...). At this point neither >>>>>> of the objects exists assuming none of the threads has yet reached >>>>>> registerObject(...). >>>>>> If one of the threads then get the writeLock, does what it needs to do >>>>>> and releases the writeLock before the second thread reaches writeLock, >>>>>> then the second thread will reach registerObject(...). As far as I can >>>>>> tell, inside registerObject(...) an SQLException will be thrown since >>>>>> the row already exists in the database, and the method >>>>>> unregisterObject(...) will be called actually unregistering a legally >>>>>> created object. >>>>>> >>>>>> Someone with more insight into the code should verify if the above is >>>>>> correct. >>>>>> >>>>>> -Jesper >>>>>> >>>>>> >>>>>> >>>>>> On tir, 2012-11-06 at 21:44 +0100, Jesper Damkjaer wrote: >>>>>>> Hi. >>>>>>> >>>>>>> I originally opened the https://jira.duraspace.org/browse/FCREPO-1024 >>>>>>> I tried back then to create tests in order to convince myself that there >>>>>>> would be no problems in removing the synchronization, but that proved >>>>>>> rather difficult. >>>>>>> As far as I remember there seemed to be some problems associated with >>>>>>> removing the synchronization, especially if two documents with the same >>>>>>> PID were ingested at the same time. >>>>>>> I'm still interested in looking in to this, and will be happy to look at >>>>>>> my old branch. >>>>>>> >>>>>>> -Jesper >>>>>>> >>>>>>> >>>>>>> On tir, 2012-11-06 at 12:46 -0500, aj...@virginia.edu wrote: >>>>>>>> There is a bit of history to this discussion: >>>>>>>> >>>>>>>> https://jira.duraspace.org/browse/FCREPO-1024 >>>>>>>> >>>>>>>> --- >>>>>>>> A. Soroka >>>>>>>> Software & Systems Engineering :: Online Library Environment >>>>>>>> the University of Virginia Library >>>>>>>> >>>>>>>> On Nov 6, 2012, at 11:53 AM, Scott Prater wrote: >>>>>>>> >>>>>>>>> Nicolas, >>>>>>>>> >>>>>>>>> That's very encouraging... this problem has been on my radar for >>>>>>>>> years. >>>>>>>>> >>>>>>>>> Not being very familiar with the details of the internals of >>>>>>>>> DefaultDOManager, I can't comment concretely on the merits of your >>>>>>>>> patch, but I would be especially interested in tests that provoke >>>>>>>>> race conditions and concurrent writes, and makes sure Fedora handles >>>>>>>>> those situations cleanly: even though PID generation is synchronized, >>>>>>>>> would it still be possible for Fedora to attempt to write the same >>>>>>>>> datastreams, provision identical values in the resource index, etc. >>>>>>>>> in different threads once the object has been created and the PID >>>>>>>>> assigned? As I dimly recall, I think the getIngestWriter method was >>>>>>>>> synchronized because there were some problems in the early days with >>>>>>>>> concurrent writes. That may be a non-issue now, though, if PID >>>>>>>>> generation is synchronized (the Akubra storage layer is designed to >>>>>>>>> handle writes more robustly). >>>>>>>>> >>>>>>>>> Another potential issue: if you're creating a hierarchical tree of >>>>>>>>> objects in parallel, and the ingest of a parent object fails: you >>>>>>>>> could be left with orphaned children. But that's something that >>>>>>>>> should be checked and handled higher up the stack, with some >>>>>>>>> transaction/rollback logic. >>>>>>>>> >>>>>>>>> -- Scott >>>>>>>>> >>>>>>>>> On 11/06/12, Nicolas Hervé wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I think I've found the main problem with massive parallel ingestion. >>>>>>>>>> >>>>>>>>>> I'm working with the last github snapshot. >>>>>>>>>> >>>>>>>>>> In org.fcrepo.server.storage.DefaultDOManager, the getIngestWriter >>>>>>>>>> method should not be synchronized as it seems there is only a single >>>>>>>>>> instance of that class for the server. The internal objects of this >>>>>>>>>> class seem to be correctly synchronized (pid generation) of new >>>>>>>>>> objects are recreated on each call (inside Translator, a new >>>>>>>>>> DODeserializer is created and the same happens inside the Validator). >>>>>>>>>> >>>>>>>>>> I've tested with FOXML ingestion and now almost all the CPUs are >>>>>>>>>> used. I've not been deeper to check that every inserted object is >>>>>>>>>> not corrupted, but after a quick look, it seems OK. I guess the same >>>>>>>>>> kind of patch could also apply on object deletion. >>>>>>>>>> >>>>>>>>>> If one of you that better understand that part could have a look, it >>>>>>>>>> seems it would be a nice patch, not too hard to test, with great >>>>>>>>>> performance improvements. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> >>>>>>>>>> Nicolas HERVE >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 28/09/2012 11:25, Nicolas Hervé wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> indeed, it seems we are exactly in the same configuration (millions >>>>>>>>>>> of DO with some metadata and external content) with almost the same >>>>>>>>>>> hardware. I've not identified the bottleneck in the massive >>>>>>>>>>> parallel ingestion process right now, but I highly suspect a >>>>>>>>>>> synchronized portion of code somewhere in the chain. I hope Edwin >>>>>>>>>>> could say more about this :-) >>>>>>>>>>> >>>>>>>>>>> For the querying of dc fields, index have to be created in the >>>>>>>>>>> Mysql schema and SQL queries are far from being optimal. Currently >>>>>>>>>>> I only patched for my own purposes (my datamodel / my queries) and >>>>>>>>>>> I bypass some code portions in the following classes : >>>>>>>>>>> >>>>>>>>>>> org.fcrepo.server.search.FieldSearchSQLImpl >>>>>>>>>>> org.fcrepo.server.search.FieldSearchResultSQLImpl >>>>>>>>>>> >>>>>>>>>>> I'm really new to Fedora Commons but, from what I understand, these >>>>>>>>>>> SQL part is quite old. Changing them for optimizations purposes >>>>>>>>>>> could imply behaviour changes for other people. That's why I don't >>>>>>>>>>> think simple patches could do the job. It would need a complete >>>>>>>>>>> refactoring. That could only be done with a global point of view on >>>>>>>>>>> the different way this classes are used in the different contexts >>>>>>>>>>> where Fedora instances are running. >>>>>>>>>>> >>>>>>>>>>> Feel free to contact me to discuss this more precisely. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> >>>>>>>>>>> Nicolas HERVE >>>>>>>>>>> +33 1 49 83 21 66 (GMT + 2) >>>>>>>>>>> >>>>>>>>>>> On 27/09/2012 18:23, Jason V wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Hi Nicolas, >>>>>>>>>>>> >>>>>>>>>>>> My name is Jason Varghese and I'm a senior developer at the New >>>>>>>>>>>> York Public Library. I think you are doing work similar to what I >>>>>>>>>>>> am presently doing based on reading some of your posts. >>>>>>>>>>>> >>>>>>>>>>>> We have a relatively large scale Fedora implementation here. We've >>>>>>>>>>>> had all the hardware in place for some time and are in the process >>>>>>>>>>>> of migrating from a large homegrown repository to a Fedora based >>>>>>>>>>>> platform. We have a single Fedora ingest machine and 3 Fedora >>>>>>>>>>>> readers. The ingest machine alone is 4 x 6 core processors w/ >>>>>>>>>>>> 128GB RAM. I'm in the process of generating about 1 million+ >>>>>>>>>>>> digital objects and attaching to each DO all the metadata (as >>>>>>>>>>>> managed content datastreams) and all the digital assets (as >>>>>>>>>>>> external content datastreams). The digital assets currently are >>>>>>>>>>>> about 183 TB of content (this is replicated at two sites). I have >>>>>>>>>>>> a multithreaded java client I wrote to accomplish the task for the >>>>>>>>>>>> Fedora ingest/DO generation and I use the Mediashelf REST API >>>>>>>>>>>> client for connectivity to Fedora. I was able to successfully >>>>>>>>>>>> ingest 10's of thousands of digital objects, but really need >>>>>>>>>>>> ensure this process performs optimally and scales for millions of >>>>>>>>>>>> objects. What bottlenecks were you able to identify when running >>>>>>>>>>>> your multithreaded ingest process? Look forward to >>>>>>>>>>>> learning/sharing experiences from this process with you and the >>>>>>>>>>>> community and possibly collaborating. Thanks >>>>>>>>>>>> >>>>>>>>>>>> Jason Varghese >>>>>>>>>>>> NYPL >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> ------------------------------------------------------------------------------ >>>>>>>>>>> Got visibility? Most devs has no idea what their production app >>>>>>>>>>> looks like. Find out how fast your code is with AppDynamics Lite. >>>>>>>>>>> http://ad.doubleclick.net/clk;262219671;13503038;y? >>>>>>>>>>> http://info.appdynamics.com/FreeJavaPerformanceDownload.html >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Fedora-commons-users mailing list >>>>>>>>>>> fedora-commons-us...@lists.sourceforge.net >>>>>>>>>>> <fedora-commons-us...@lists.sourceforge.net> >>>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users >>>>>>>>>>> >>>>>>>>> -- >>>>>>>>> -- >>>>>>>>> Scott Prater >>>>>>>>> Library, Instructional, and Research Applications (LIRA) >>>>>>>>> Division of Information Technology (DoIT) >>>>>>>>> University of Wisconsin - Madison >>>>>>>>> pra...@wisc.edu >>>>>>>>> >>>>>>>>> ------------------------------------------------------------------------------ >>>>>>>>> LogMeIn Central: Instant, anywhere, Remote PC access and management. >>>>>>>>> Stay in control, update software, and manage PCs from one command >>>>>>>>> center >>>>>>>>> Diagnose problems and improve visibility into emerging IT issues >>>>>>>>> Automate, monitor and manage. Do more in less time with Central >>>>>>>>> http://p.sf.net/sfu/logmein12331_d2d >>>>>>>>> _______________________________________________ >>>>>>>>> Fedora-commons-developers mailing list >>>>>>>>> Fedora-commons-developers@lists.sourceforge.net >>>>>>>>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-developers >>>>>>>> >>>>>>>> ------------------------------------------------------------------------------ >>>>>>>> LogMeIn Central: Instant, anywhere, Remote PC access and management. >>>>>>>> Stay in control, update software, and manage PCs from one command >>>>>>>> center >>>>>>>> Diagnose problems and improve visibility into emerging IT issues >>>>>>>> Automate, monitor and manage. Do more in less time with Central >>>>>>>> http://p.sf.net/sfu/logmein12331_d2d >>>>>>>> _______________________________________________ >>>>>>>> Fedora-commons-developers mailing list >>>>>>>> Fedora-commons-developers@lists.sourceforge.net >>>>>>>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-developers >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> ------------------------------------------------------------------------------ >>>>>>> LogMeIn Central: Instant, anywhere, Remote PC access and management. >>>>>>> Stay in control, update software, and manage PCs from one command center >>>>>>> Diagnose problems and improve visibility into emerging IT issues >>>>>>> Automate, monitor and manage. Do more in less time with Central >>>>>>> http://p.sf.net/sfu/logmein12331_d2d >>>>>>> _______________________________________________ >>>>>>> Fedora-commons-developers mailing list >>>>>>> Fedora-commons-developers@lists.sourceforge.net >>>>>>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-developers >>>>>> >>>>>> >>>>>> >>>>>> ------------------------------------------------------------------------------ >>>>>> LogMeIn Central: Instant, anywhere, Remote PC access and management. >>>>>> Stay in control, update software, and manage PCs from one command center >>>>>> Diagnose problems and improve visibility into emerging IT issues >>>>>> Automate, monitor and manage. Do more in less time with Central >>>>>> http://p.sf.net/sfu/logmein12331_d2d >>>>>> _______________________________________________ >>>>>> Fedora-commons-developers mailing list >>>>>> Fedora-commons-developers@lists.sourceforge.net >>>>>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-developers >>>>> >>>>> ------------------------------------------------------------------------------ >>>>> LogMeIn Central: Instant, anywhere, Remote PC access and management. >>>>> Stay in control, update software, and manage PCs from one command center >>>>> Diagnose problems and improve visibility into emerging IT issues >>>>> Automate, monitor and manage. Do more in less time with Central >>>>> http://p.sf.net/sfu/logmein12331_d2d >>>>> _______________________________________________ >>>>> Fedora-commons-developers mailing list >>>>> Fedora-commons-developers@lists.sourceforge.net >>>>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-developers >>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> LogMeIn Central: Instant, anywhere, Remote PC access and management. >>>> Stay in control, update software, and manage PCs from one command center >>>> Diagnose problems and improve visibility into emerging IT issues >>>> Automate, monitor and manage. Do more in less time with Central >>>> http://p.sf.net/sfu/logmein12331_d2d >>>> _______________________________________________ >>>> Fedora-commons-developers mailing list >>>> Fedora-commons-developers@lists.sourceforge.net >>>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-developers >>> >>> ------------------------------------------------------------------------------ >>> LogMeIn Central: Instant, anywhere, Remote PC access and management. >>> Stay in control, update software, and manage PCs from one command center >>> Diagnose problems and improve visibility into emerging IT issues >>> Automate, monitor and manage. Do more in less time with Central >>> http://p.sf.net/sfu/logmein12331_d2d >>> _______________________________________________ >>> Fedora-commons-developers mailing list >>> Fedora-commons-developers@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-developers >> >> >> >> >> ------------------------------------------------------------------------------ >> LogMeIn Central: Instant, anywhere, Remote PC access and management. >> Stay in control, update software, and manage PCs from one command center >> Diagnose problems and improve visibility into emerging IT issues >> Automate, monitor and manage. Do more in less time with Central >> http://p.sf.net/sfu/logmein12331_d2d >> _______________________________________________ >> Fedora-commons-developers mailing list >> Fedora-commons-developers@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/fedora-commons-developers > > > > > ------------------------------------------------------------------------------ > LogMeIn Central: Instant, anywhere, Remote PC access and management. > Stay in control, update software, and manage PCs from one command center > Diagnose problems and improve visibility into emerging IT issues > Automate, monitor and manage. Do more in less time with Central > http://p.sf.net/sfu/logmein12331_d2d > _______________________________________________ > Fedora-commons-developers mailing list > Fedora-commons-developers@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/fedora-commons-developers ------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_d2d_nov _______________________________________________ Fedora-commons-developers mailing list Fedora-commons-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/fedora-commons-developers
