Hey Mo and Prof Mattmann, I will try to crawl the 3 websites in the homework tonight (NASA AMD, NSF ACADIS and NSIDC Arctic Data Explorer). I will let you know what's going on.
Is memory an issue? My vagrant only has 512MB of memory. Regards, Shuo Li On Fri, Feb 13, 2015 at 10:25 AM, Mattmann, Chris A (3980) < chris.a.mattm...@jpl.nasa.gov> wrote: > Hi Shuo, > > Thanks for your email. I wonder if using selenium grid would > help? > > Please see this plugin: > > https://github.com/momer/nutch-selenium-grid-plugin > > > I’m CC’ing Mo the author of the plugin to see if he experienced > this while running the original selenium plugin - Mo did using > selenium grid help the issue that Shuo is experiencing below? > > Mo: are you cool with portion the grid plugin, or if Lewis or > I do it to trunk (with full credit to you of course?) > > Cheers, > Chris > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > Chris Mattmann, Ph.D. > Chief Architect > Instrument Software and Science Data Systems Section (398) > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA > Office: 168-519, Mailstop: 168-527 > Email: chris.a.mattm...@nasa.gov > WWW: http://sunset.usc.edu/~mattmann/ > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > Adjunct Associate Professor, Computer Science Department > University of Southern California, Los Angeles, CA 90089 USA > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > > > > > -----Original Message----- > From: Shuo Li <sli...@usc.edu> > Reply-To: "dev@nutch.apache.org" <dev@nutch.apache.org> > Date: Friday, February 13, 2015 at 10:12 AM > To: "dev@nutch.apache.org" <dev@nutch.apache.org> > Subject: Vagrant Crushed When using Nutch-Selenium > > >Hey guys, > > > > > >I'm trying to use Nutch-Selenium to crawl > >nutch.apache.org <http://nutch.apache.org>. However, my vagrant seems > >crushed after a few minutes. I forced it to shut down and it turns out it > >only crawled 59 websites. My nutch version is 1.10 and my OS is Ubuntu > >Trusty, 14.04. > > > > > >Is there anything I can provide to you guys? Or is there anybody have the > >same issue? Or 59 websites is the complete crawling? > > > > > >Any suggestion would be appreciated. > > > > > >Regards, > >Shuo Li > > > >
