[Nutch-dev] Fwd: [Collex] application#index (ActionController::RoutingError) "no route found to match \"/nines/ escape(document.title) u, \" with {:method=>:get}"

Please reply to me directly as well, as I'm not on the nutch-dev list regularly. I'm curious ... Googlebot, Yahoo Slurp, and now CazoodleBot (based on Nutch) are hitting our site at http://www.nines.org and I get all sorts of invalid links crawled. Is our site doing something wrong in our

[Nutch-dev] [jira] Commented: (NUTCH-134) Summarizer doesn't select the best snippets

[ http://issues.apache.org/jira/browse/NUTCH-134?page=comments#action_12361351 ] Erik Hatcher commented on NUTCH-134: Byron - It's under contrib/highlighter. For Nutch, which uses Lucene's trunk version, you'll want to build the H

[Nutch-dev] Re: GNU Getopt

On Dec 20, 2005, at 3:02 AM, Andrzej Bialecki wrote: I believe there is a similar library in Jakarta Commons, I don't know if it provides similar functionality...? Jakarta Commons CLI: http://jakarta.apache.org/commons/cli/ ---

[Nutch-dev] TrustRank (was Re: "Something is Wrong with G oogle’s Mathematical Model")

On Dec 16, 2005, at 4:09 PM, Fredrik Andersson wrote: While on the topic, during the "Bourbon update" earlier this year, rumors were flying around about the "TrustRank" algorithm, which involved some human input on validating credible sources of data on the web. There's a paper from Stanfor

[Nutch-dev] Re: [VOTE] Commiter access for Stefan Groschupf

+1 On Dec 16, 2005, at 4:50 PM, Andrzej Bialecki wrote: Hi, During the past year and more Stefan participated actively in the development, and contributed many high-quality patches. He's been spending considerable effort on addressing many issues in JIRA, and proposing fixes and improvements.

Re: [Nutch-dev] RE: [proposal] Generic Markup Language Parser

On 24 Nov 2005, at 23:49, Chris Mattmann wrote: Dublin core may is good for semantic web, but not for a content storage. I completely disagree with that. Me too. In fact, I think many people would disagree with that in fact. Dublin core is a "standard" metadata model for electronic reso

[Nutch-dev] Re: Lucene or Nutch

Yes, Lucene is the best fit for what you're after. Nutch is built on Lucene, and adds web crawling on top. You don't need a web crawler, so using Lucene directly is the best fit - of course you'll have to write code to integrate Lucene. Erik On 9 Nov 2005, at 08:48, Klaus wrote: H

[Nutch-dev] Re: Javacc

What version of Ant are you using and what version of Lucene? The latest trunk version of Lucene has gone back to using the task in Ant, which is a facade that handles all the various versions of JavaCC, so my hunch is that your Ant distribution needs to be updated. Erik On 6 Nov 20

[Nutch-dev] Re: svn commit: r314958 - in /lucene/nutch/trunk/site: about.html bot.html credits.html i18n.html index.html index.pdf issue_tracking.html linkmap.html mailing_lists.html tutorial.html ver

Thanks Doug!!! On Oct 12, 2005, at 12:32 PM, [EMAIL PROTECTED] wrote: Author: cutting Date: Wed Oct 12 09:31:33 2005 New Revision: 314958 URL: http://svn.apache.org/viewcvs?rev=314958&view=rev Log: Use mirrors for downloads. --- This SF.N

[Nutch-dev] Re: nutch downloads

Joshua, We have received your message. I'm only remotely involved with Nutch, so I'm prodding other committers to Nutch to please update the links to take advantage of the mirroring system in place. Please - someone reply back volunteering to correct this ASAP. Erik On Oct 11, 2005,

[Nutch-dev] Re: work on Nutch made Index with Lukes HighFreqTerms

Nils, Your message is best directed to java-user@lucene.apache.org (please subscribe before sending to this address). Erik On Sep 5, 2005, at 3:22 PM, Nils Hoeller wrote: Hi, I now have implemented the Top Ten Term Search I have asked about before. I just need to filter the stuff wi

Re: [Nutch-dev] Re: Another NDFS question

What you've just described, Doug, is WebDAV! There is an implementation of it built into Tomcat, but a more full-featured version is Slide - http://jakarta.apache.org/slide/ . There is also JSR (#170) for a content repository, being implemented open-source as Jackrabbit: http://incubator.

[Nutch-dev] Re: junit test failed

This has nothing to do with the version of Ant. JUnit's JAR file needs to be in ANT_HOME/lib for to work. Erik On Aug 28, 2005, at 9:45 PM, Fuad Efendi wrote: Check version of ANT! Line 173: nutch\trunk\build.xml:173: Could not create task or type of type: junit. Probably,

[Nutch-dev] Re: Fetcher for constrained crawls

Kelvin, Big +1!!! I'm working on focused crawling as well, and your work fits well with my needs. An implementation detail - have you considered using HiveMind rather than Spring? This would be much more compatible license-wise with Nutch and be easier to integrate into the ASF reposito

[Nutch-dev] Outlink metadata?

First a question about the current behavior... does Nutch adhere to the conventions? If so, where is that coded? On a related note, it seems carrying metadata around on Outlink would be beneficial, not just anchor text and URL. For example, my application will crawl HTML sites with a HEA

[Nutch-dev] Re: Merge Lucene to Nutch

Nutch simply uses the Lucene JAR file. Upgrading Nutch to use a new Lucene release would involve replacing the JAR file with the new version, and depending on the changes to Lucene itself it may involve rebuilding indexes (to ensure normalization factors and such changes are incorporated),

[Nutch-dev] Re: 0.7-dev, the search scoring

On Jul 28, 2005, at 8:28 AM, Fredrik Andersson wrote: Also, are there any "secret" WIKIs, mailinglists, forums or similar for the 0.7 development? No. All discussions are in the open right here on nutch-dev and nutch-user. Erik

[Nutch-dev] Re: Corrections to README.txt

I've corrected this and committed it. Thanks for letting us know. Erik On Jul 27, 2005, at 2:59 PM, Hasan Diwan wrote: I just looked at README.txt (revision 225574) and noted the URLs in it need to be updated from the nutch.org to the new ones. Cheers, Hasan Diwan <[EMAIL PROTECTED]>

[Nutch-dev] Re: Information extraction

Further on the information extraction idea, consider what the SIMILE team at MIT are doing... http://simile.mit.edu The lower-case semantic web is gaining a lot of momentum these days, and I'm a strong proponent and student of it at the moment. Scraping rich information from a site is cert

Re: [Nutch-dev] getDiscriptor

On Jul 21, 2005, at 4:02 PM, Jérôme Charron wrote: And also PluginRepository.dependencyIsAvailabel - rename, or deprecate and correct? Erik, what a good code reviewer you are! You know what I think about deprecated methods (If the probability to be used outside of Nutch, then must be depre

[Nutch-dev] Re: parser plugin lifecycle

On Jul 21, 2005, at 11:46 AM, Stefan Groschupf wrote: In short each plugin can have a plugin class you can define in the plugin deployment descriptor. This plugin class has life cycle methods that will be invoked until start up to create a db connection and until shutdown to close a db conn

Re: [Nutch-dev] getDiscriptor

And also PluginRepository.dependencyIsAvailabel - rename, or deprecate and correct? On Jul 21, 2005, at 1:56 PM, Erik Hatcher wrote: Another misspelling Extension.getDiscriptor/setDiscriptor. Shall I simply rename it or does it need to be deprecated for a while to transition

[Nutch-dev] getDiscriptor

Another misspelling Extension.getDiscriptor/setDiscriptor. Shall I simply rename it or does it need to be deprecated for a while to transition? Erik --- SF.Net email is sponsored by: Discover Easy Linux Migration Strategies from

[Nutch-dev] Re: parser plugin lifecycle

On Jul 21, 2005, at 11:46 AM, Stefan Groschupf wrote: Hi Erik, Check: Introduction to the nutch plug-in architecture http://www.media-style.com/index.jsp?folderPK=422 It gives some basic understanding of the plugin life cycle. In short each plugin can have a plugin class you can define in the

[Nutch-dev] parser plugin lifecycle

How could a Parser implementation do some cleanup at the end of the process? I'm new to the plugin lifecycle - is this currently possible? I have an RDF parser plugin that needs to open a connection to an external RDF engine and I need to do that only one time and then close that connectio

[Nutch-dev] Re: svn commit: r220056 - /lucene/nutch/trunk/src/test/org/apache/nutch/plugin/TestPluginSystem.java

On Jul 21, 2005, at 10:22 AM, Erik Hatcher wrote: On Jul 21, 2005, at 9:21 AM, Jérôme Charron wrote: For grins I tried to see if I had commit access to fix the misspelling myself. Lo and behold I do! I hope I didn't step on any toes by committing this - if so let me know and I'

[Nutch-dev] Re: svn commit: r220056 - /lucene/nutch/trunk/src/test/org/apache/nutch/plugin/TestPluginSystem.java

On Jul 21, 2005, at 9:21 AM, Jérôme Charron wrote: For grins I tried to see if I had commit access to fix the misspelling myself. Lo and behold I do! I hope I didn't step on any toes by committing this - if so let me know and I'll be more patient and submit patches. I'm a newbie to Nutch and de

[Nutch-dev] Fwd: svn commit: r220056 - /lucene/nutch/trunk/src/test/org/apache/nutch/plugin/TestPluginSystem.java

For grins I tried to see if I had commit access to fix the misspelling myself. Lo and behold I do! I hope I didn't step on any toes by committing this - if so let me know and I'll be more patient and submit patches. I'm a newbie to Nutch and definitely don't want to step in to committing

[Nutch-dev] API misspelling?

ExtensionPoint.getExtentens() - is this intentional or a misspelling? Erik --- SF.Net email is sponsored by: Discover Easy Linux Migration Strategies from IBM. Find simple to follow Roadmaps, straightforward articles, informative Webcasts

[Nutch-dev] bin/nutch issue - on Mac OS X

I'm getting "expr: syntax error" when running all bin/nutch commands. It comes from this line: if expr match `uname` 'CYGWIN*' > /dev/null; then should this be modified to be this instead: if expr `uname` : 'CYGWIN*' > /dev/null; then That works for me, but I'm not running Cygwin.

Re: [Nutch-dev] Re: ESP - Ethics search protocol for internet search engines.

On Jul 10, 2005, at 3:58 PM, Bernhard Fastenrath wrote: I mentioned CC because it parses embedded RDF data that is making a statement about the resources available from that page. I thought the connection between making statements about ethics could be made similarly. That is concei

[Nutch-dev] Re: ESP - Ethics search protocol for internet search engines.

On Jul 10, 2005, at 9:30 AM, Bernhard Fastenrath wrote: Erik Hatcher wrote: It would seem that RDF and the Creative Commons plugin would be very related to what you're doing. I'm curious if you've given RDF consideration in this and how that would factor into it?

[Nutch-dev] Re: ESP - Ethics search protocol for internet search engines.

It would seem that RDF and the Creative Commons plugin would be very related to what you're doing. I'm curious if you've given RDF consideration in this and how that would factor into it? Erik On Jul 9, 2005, at 8:22 AM, Bernhard Fastenrath wrote: I'm working on an ethics search pro

[Nutch-dev] Re: [VOTE] new Nutch committers

+2 On Jun 8, 2005, at 4:09 PM, Doug Cutting wrote: I propose that we add Jérôme Charron and Piotr Kosiorowski as Nutch committers. Both Jérôme and Piotr have contributed many high quality patches to Nutch, giving me confidence that they can be trusted to directly maintain it. Formally,

[Nutch-dev] Re: [EMAIL PROTECTED] Mailinglist

I'm getting multiple messages to the list. I'm not showing as subscribed to the sourceforge list, but I get 3 copies of each Nutch message. I need to get that straightened out sometime. Erik On Apr 20, 2005, at 1:07 PM, Doug Cutting wrote: Michael Wechner wrote: Sorry if this might be

[Nutch-dev] Re: Nutch / CGI

I do not believe that the file formats are identical between Java Lucene and PLucene. So be careful in that regard. Another option to access Lucene indexes from scripting languages is PyLucene, which is very robust and guaranteed to be compatible with Java Lucene (since it is built directly fr

[Nutch-dev] Re: [Nutch-cvs] svn commit: r158838 - incubator/nutch/trunk/site/i18n.html incubator/nutch/trunk/site/i18n.pdf

Hari - you seem to be right. I really thought there was a way to configure this on the server-side - I mixed up the client and server configurations in this regard. Perhaps its possible to do this using a pre-commit hook though? Erik On Mar 24, 2005, at 1:26 AM, Hari Kodungallur wrote:

[Nutch-dev] Re: [Nutch-cvs] svn commit: r158838 - incubator/nutch/trunk/site/i18n.html incubator/nutch/trunk/site/i18n.pdf

On Mar 23, 2005, at 5:04 PM, Andrzej Bialecki wrote: [EMAIL PROTECTED] wrote: Author: cutting Date: Wed Mar 23 13:56:22 2005 New Revision: 158838 URL: http://svn.apache.org/viewcvs?view=rev&rev=158838 Log: Forgot to add these rendered i18n docs... Added: incubator/nutch/trunk/site/i18n.html

[Nutch-dev] Fwd: what has happened to www.apache.org?

*nudge nudge* :) Erik Begin forwarded message: From: Paul Querna <[EMAIL PROTECTED]> Date: March 22, 2005 7:19:45 PM EST To: "Noel J. Bergman" <[EMAIL PROTECTED]> Cc: Stas Bekman <[EMAIL PROTECTED]>, [EMAIL PROTECTED] Subject: Re: what has happened to www.apache.org? Noel J. Bergman wrote:

[Nutch-dev] Re: [Nutch-cvs] nutch/src/java/net/nutch/mapReduce MapOutputLocation.java,NONE,1.1

Shouldn't you be committing to the Apache svn repo now? Or how are you managing it? Erik On Feb 2, 2005, at 4:36 PM, Doug Cutting wrote: Update of /cvsroot/nutch/nutch/src/java/net/nutch/mapReduce In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv12267 Added Files: MapOutputL

Re: [Nutch-dev] Results Syndication Feed

On Dec 30, 2004, at 1:00 PM, Hasan Diwan wrote: Are you asking about how HTML 's behave? It's non-sensical, but checkboxes by do not submit anything if they are unchecked and their value if they are checked. The value attribute is listed as optional. Hence I'm wondering what the default value is.

Re: [Nutch-dev] Results Syndication Feed

On Dec 30, 2004, at 12:32 PM, Hasan Diwan wrote: On Tue, 28 Dec 2004 08:35:28 -0800, Doug Cutting <[EMAIL PROTECTED]> wrote: A servlet that uses NutchBean should do most of what's required. Since I'm thinking of making Rss feeds optional and default to off, I need to add a checkbox to search.jsp i

[Nutch-dev] Lucene in Action e-book now available

The long awaited Lucene in Action e-book is now available at Manning's site: http://www.manning.com/hatcher2 Manning also put lots of other goodies there, the table of contents, "about this book", preface, the foreward from Doug Cutting himself (thanks Doug!!!), and a couple of sample chapt

Re: [Nutch-dev] wiki gone; backups

On Nov 23, 2004, at 9:38 PM, Nick Lothian wrote: There was a proposal for the incubation of a search engine called Locust on the [EMAIL PROTECTED] list last week. See I corresponded privately with the person posting

Re: [Nutch-dev] wiki gone; backups

Just a thought - Another option would be to recreate the Nutch wiki under the Apache Jakarta Lucene wiki. Hopefully Nutch will be an Apache project sometime in the near future. Erik On Nov 22, 2004, at 8:41 AM, Doug Cutting wrote: It looks like someone removed Nutch's wiki data yesterday

[Nutch-dev] wiki down?

The Nutch wiki appears to be down. What happened? Could someone bring it back to life? Thanks, Erik --- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover

Re: [Nutch-dev] JIRA was VOTE: licenses

I don't see a pressing need to get JIRA installed for Nutch or Lucene prior to the aggregation at Apache. It'd probably be best to do this all at once when we set up search.apache.org. Erik On Nov 11, 2004, at 9:38 AM, [EMAIL PROTECTED] wrote: We can also upgrade to JIRA as an issue tra

Re: [Nutch-dev] VOTE: licenses

I'll add my thoughts to this thread. It has been discussed for quite some time to bring Lucene to a top-level Apache project. I plan on assisting with this effort starting in December. Bringing all the ports of Lucene together will allow us to put together a comprehensive test suite to show

Re: [Nutch-dev] Nutch

On Oct 8, 2004, at 9:31 AM, McGovern, James F (HTSC, IT) wrote: Came across your project and thought I could provide assistance in the way of getting the word out. As an author and speaker at industry conferences, I have the ability to spread the word. Let me know how I can help. James - welcome

[Nutch-dev] Re: [Nutch-cvs] nutch CHANGES.txt,1.27,1.28 default.properties,1.25,1.26

On Aug 30, 2004, at 4:43 PM, Doug Cutting wrote: plugin.file=net.nutch.protocol.file* plugin.html=net.nutch.parse.html* + plugin.msword=net.nutch.parse.mp3* shouldn't this be plugin.mp3? Erik --- This SF.Net email is sponsored by BEA

[Nutch-dev] Re: [Nutch-cvs] nutch/conf nutch-default.xml,1.43,1.44

On Jul 16, 2004, at 8:11 PM, Andrzej Bialecki wrote: Modified Files: nutch-default.xml + + earcher.hostgrouping.rawhits.factor Shouldn't this be "searcher"?? + 2 + + A factor that is used to determine the number of raw hits + initially fetched, before host grouping is done. +

Re: [Nutch-dev] local search

On Jun 8, 2004, at 5:25 PM, Doug Cutting wrote: Erik Hatcher wrote: For the upcoming Lucene in Action (http://www.manning.com/hatcher2) I created a custom sort (something new in Lucene 1.4) comparator that sorts based on 2-dimension distance [d = sqrt( (x1-x2)^2 + (y1-y2)^2)]. The new sort

Re: [Nutch-dev] local search

On Jun 8, 2004, at 10:56 AM, Matthias Jaekle wrote: Searching for restaurant in the area 50 km around Stuttgart is done on the following page: http://www.iventax.de:8080/search.jsp? query=restaurant&hitsPerPage=50&north=48.780&east=9.180&radius=50&reduc tion=2 This is quite cool! Are the result

Re: [Nutch-dev] Re-org CVS - Make paths off root for code (instead of src.java.*)

On Jun 6, 2004, at 10:21 AM, Byron Miller wrote: I just figured since every other project i personally do i base everything off the class path from the root of my source dir and that is what i was asking if nutch could do so i don't have to customize all of my editors from the get go (as the src.ja

Re: [Nutch-dev] web ui: no improvements :(

Not at all surprising, since JSP's compile into servlets :)) Erik On Jun 4, 2004, at 11:32 AM, Stefan Groschupf wrote: Hi, I'm sorry to say but after a week I'm sure the speed improvements by using servlets are too minimal. ;( One beer for Doug on my bill. ;-) I had done some stress tests

[Nutch-dev] Fwd: [Nutch-cvs] nutch/src/plugin build.xml,1.3,1.4

One build option you might want to employ with the plugins is to borrow what I did for the jakarta-lucene-sandbox repository under the "contributions" directory. I use the new task to crawl all directories (can be open-ended or a precisely specified list, your choice) and run a build file. T

Re: [Nutch-dev] improve web-ui

On May 28, 2004, at 5:14 PM, Stefan Groschupf wrote: Well the second edition of your book could include some hints how to use ant inside of jelly script. ;) Jelly is an abomination. The creator even apologizes: http://radio.weblogs.com/0112098/2004/03/26.html#a472 People need still ant kn

Re: [Nutch-dev] improve web-ui

On May 28, 2004, at 10:17 AM, . . wrote: So you are saying Resin works with nutch and it work fine :) No, I didn't say that. I'm only saying that Resin is well-known as *fast*. Erik --- This SF.Net email is sponsored by: Oracle 10g Get

Re: [Nutch-dev] improve web-ui

On May 28, 2004, at 3:17 PM, Stefan Groschupf wrote: Well, "in case" we would use maven we would have the solution ready yet since it is build in. But since we ;-) Serious I strongly think nutch need a maven build, but we will discuss this on a other time. *yuck* :) http://howardlew

Re: [Nutch-dev] improve web-ui

On May 28, 2004, at 4:25 AM, Stefan Groschupf wrote: At least it think that is not question of java version 1.3 or 1.4 more a question of the servlet specification and the implementation itself. For example I person have the experience that jetty is much much faster then tomcat. Do you have mor

[Nutch-dev] TheServerSide Symposium Nutch presentation

Doug received quite a nice review of his TSSS presentation on Nutch: --- This SF.Net email is sponsored by Sleepycat Software Learn developer strategies Cisco, Motorola, Ericsson

Re: [Nutch-dev] (no subject)

Not exactly Nutch uses its own query parser, right? I don't think it supports the same syntax, but a smaller set of operations. Also, maybe Roy was talking about a different type of "proximity". Doug added a very slick span feature to Lucene 1.4 that can do proximity of phrase queries, no

[Nutch-dev] Fwd: Google Web Alert - erik hatcher

Nutch needs *this* feature. Wow! Kinda cool to have my own personal spy keeping tabs on me behind the scenes. Erik Begin forwarded message: From: [EMAIL PROTECTED] Date: April 12, 2004 9:46:56 PM EDT To: [EMAIL PROTECTED] Subject: Google Web Alert - erik hatcher THIS WEEK Richmond Times

Re: [Nutch-dev] compare Google & Yahoo rankings

On Feb 26, 2004, at 4:59 PM, Doug Cutting wrote: FYI, here's a cool tool to visually compare rankings: http://www.langreiter.com/exec/yahoo-vs-google.html?q=nutch For most of the searches I've tried they agree on the first result, and then things differ, usually by quite a bit. The Yahoo results

Re: [Nutch-dev] indexing mail archives

Eyebrowse uses Lucene to index/search e-mail lists: http://eyebrowse.tigris.org/ On Jan 28, 2004, at 2:15 PM, Stefan Groschupf wrote: HI, Lucene is often used to index mail archive. I do not know if there is any ready to use solution using lucence, but it should take nothing more then a half