Hi, On 5/29/07, Nicolás Lichtmaier <[EMAIL PROTECTED]> wrote: > > > Which job causes the problem? Perhaps, we can find out what keeps > > creating a conf object over and over. > > > > Also, I have tried what you have suggested (better caching for plugin > > repository) and it really seems to make a difference. Can you try with > > this patch(*) to see if it solves your problem? > > > > (*) http://www.ceng.metu.edu.tr/~e1345172/plugin_repository_cache.patch > > Some comments about you patch. The approach seems nice, you only check > the parameters that affect plugin loading. But have in mind that the > plugin themselves will configure themselves with many other parameters, > so to keep things safe there should be a PluginRepository for each set > of parameters (including all of them). Besides, remember that CACHE is a > WeakHashMap, you are creating ad-hoc PluginProperty objects as keys, > something doesn't loook right... the lifespan of those objects will be > much shorter than you require, perhaps you should be using > SoftReferences instead, or a simple LRU (LinkedHashMap provides that > simply) cache.
My patch is just a draft to see if we can create a better caching mechanism. There are definitely some rough edges there:) I don't really worry about WeakHashMap->LinkedHashMap stuff. But your approach is simple and should be faster so I guess it's OK. You are right about per-plugin parameters but I think it will be very difficult to keep PluginProperty class in sync with plugin parameters. I mean, if a plugin defines a new parameter, we have to remember to update PluginProperty. Perhaps, we can force plugins to define configuration options it will use in, say, its plugin.xml file, but that will be very error-prone too. I don't want to compare entire configuration objects, because changing irrevelant options, like fetcher.store.content shouldn't force loading plugins again, though it seems it may be inevitable.... > > Anyway, I'll try to build my own Nutch to test your patch. > > Thanks! > > -- Doğacan Güney ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ Nutch-developers mailing list Nutch-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nutch-developers
