John,
You may have a multi cpu engine?
Am 18.07.2004 um 22:17 schrieb [EMAIL PROTECTED]:
Hi, Stefan,
On Sat, Jul 17, 2004 at 11:52:47PM +0200, Stefan Groschupf wrote:
Do you use Extension.getExtensionInstance() or / and
PluginRepository.getPluginInstance(PluginDescriptor pDescriptor) to
access the plugin?
Or you may be create own instances of the HttpPlugin.java?
As you can see in PluginRepository.getPluginInstance(.. ) the plugin
instance is cached.
I have tried all those things and the problem persists.
May you discover a bug and we need to synchronized access the plugin
instance cache in the PluginRepository.
I would say there is the problem...
Your code is ok. Can you commit it to the cvs and provide a unit test,
i will try to fix the problem immediately.
The easiest may be
(1) check out current cvs
(2) add HttpPlugin.java to
./nutch/src/plugin/protocol-http/src/java/net/nutch/protocol/http/
(3) overwrite ./nutch/src/plugin/protocol-http/plugin.xml using my
plugin.xml
Both HttpPlugin.java and plugin.xml are in my previous email.
It may not be a good idea to commit them for trouble-shooting purpose.
Or for the best, I suggest adding a multi-threaded test in
./src/test/net/nutch/plugin/TestPluginSystem.java
It will make the problem evident and provide further assurance.
Multithreading is the primary usage mode of plugins in nutch.
Thanks,
John
Am 17.07.2004 um 23:54 schrieb [EMAIL PROTECTED]:
On Tue, Jul 13, 2004 at 11:56:16PM +0200, Stefan Groschupf wrote:
(1) in protocol-ftp, I would like to keep per-host connection
PERSISTENT as long as the fetcher is hitting the same host.
This can be extremely useful for ftp sites that are expensive for
initial logins. In pre-plugin nutch, it was archived by passing
handler
ftp.client from Ftp.java to FtpResponse.java. However, with new
plugin
system,
this object seemly gets garbage collected each time a url is
visited,
i.e.,
connection has to be reinitiated for next url. A huge headache.
That is why there is a Plugin class you normally need to implement
within a plugin. Doug later remove this requirement.
Until runtime it exist only one instance of the object so it is a
kind
of singleton.
Your cache should be hosted or accessed over your plugin Plugin.java
implementation.
Take a look in the snmp plugin i had submitted it iluustrate what
you
need to do. Dont miss to setup the plugin class in the plugin.xml
manifest file.
Hi, Stefan,
Using current cvs tree, I added in an HttpPlugin.java and modified
protocol-http's plugin.xml accordingly. Both are attached at the end
of this message. Nothing else in source tree was changed.
I then tried to run the fetcher a few times, using multiple threads,
on a small segment with a dozen of urls. Its behavior is puzzling:
(1) There is no consistent number of start up of HttpPlugin.java,
as indicated by logging messages such as:
040717 135633 11 start up Plugin: 11 protocol-http
040717 135633 12 start up Plugin: 12 protocol-http
......
For runs using 4 threads, I observed HttpPlugin.java
started any number btween 1-4 times.
The same is true for runs using 6, 8, 10 threads.
(2) Number of "shutdown" of HttpPlugin.java could be none or
any number up to num_threads minus 1.
Assuming HttpPlugin.java (Plugin.java) is of singleton pattern,
wouldn't we
expect its startup or shutdown only once?
Or am I missing something?
John
-------------------------------------------------------
This SF.Net email is sponsored by BEA Weblogic Workshop
FREE Java Enterprise J2EE developer tools!
Get your free copy of BEA WebLogic Workshop 8.1 today.
http://ads.osdn.com/?ad_id=4721&alloc_id=10040&op=click
_______________________________________________
Nutch-developers mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
---------------------------------------------------------------
enterprise information technology consulting
open technology: http://www.media-style.com
open discussion: http://www.text-mining.org
open thoughts: http://www.find23.net
-------------------------------------------------------
This SF.Net email is sponsored by BEA Weblogic Workshop
FREE Java Enterprise J2EE developer tools!
Get your free copy of BEA WebLogic Workshop 8.1 today.
http://ads.osdn.com/?ad_id=4721&alloc_id=10040&op=click
_______________________________________________
Nutch-developers mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
