Hi, We will have to crawl around 10 web sites with different configurations in parallel. In our current set up(5 node hadoop), we need to have 10 separate .job file with different configurations. Is there anyway we can use single nutch job file with different configurations in parallel in hadoop cluster.
It looks NUTCH_CONF_DIR is effective only in local mode as per the post http://www.mail-archive.com/[email protected]/msg01647.html. Has anyone tried this already? Please let us know. Senthil -- View this message in context: http://lucene.472066.n3.nabble.com/Nutch-2-1-Having-multiple-different-configurations-for-single-Nutch-job-in-deploy-distributed-mode-tp4104604.html Sent from the Nutch - User mailing list archive at Nabble.com.
