[ https://issues.apache.org/jira/browse/NUTCH-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17638210#comment-17638210 ]
Lucas Pauchard commented on NUTCH-2975: --------------------------------------- Hello Nagel, Thank you for the hint about the time. I found that some of our VMs were ahead of time. Once I have resynchronize the time on these VMs, the issue didn't happen again. I will therefore close this issue. Thanks a lot. > Generate 0 partition when used with sitemap > ------------------------------------------- > > Key: NUTCH-2975 > URL: https://issues.apache.org/jira/browse/NUTCH-2975 > Project: Nutch > Issue Type: Bug > Components: generator, sitemap > Affects Versions: 1.16 > Environment: Hadoop 3.1.1 > Reporter: Lucas Pauchard > Priority: Major > > *Issue* > We are facing strange issue since we have updated our Proxmox from 7.2-4 to > 7.2-11 which host the VMs/containers used for our Hadoop cluster. > When we are using the sitemap component to add URLs, the generator process > doesn't work. It generates 0 partition. > But if we call a second time the generator process, this time the generator > actually create a partition segment. > It happens only when we use the sitemap process. If we use only the Injector > process, this issue doesn't happen. > I checked the logs and the generator just seems to find no record in the > crawldb. It is like the crawldb wasn't available or the files are locked. > *Here is the command used :* > Sitemap : > {code} > hadoop jar <job> org.apache.nutch.crawl.Generator crawl_000_111/crawldb > crawl_000_111/segment -numFetchers 2 -mapper=1 -reducer=1 -noFilter -noNorm > -force > {code} > It returns as expected : > {panel:title=Sitemap output} > 2022-11-23 10:37:22,194 INFO util.SitemapProcessor: SitemapProcessor: Total > records rejected by filters: 0 > 2022-11-23 10:37:22,195 INFO util.SitemapProcessor: SitemapProcessor: Total > sitemaps from HostDb: 0 > 2022-11-23 10:37:22,195 INFO util.SitemapProcessor: SitemapProcessor: Total > sitemaps from seed urls: 1 > 2022-11-23 10:37:22,196 INFO util.SitemapProcessor: SitemapProcessor: Total > failed sitemap fetches: 0 > 2022-11-23 10:37:22,196 INFO util.SitemapProcessor: SitemapProcessor: Total > new sitemap entries added: 151 > {panel} > Generetor : > {code} > hadoop jar <job> org.apache.nutch.crawl.Generator crawl_000_111/crawldb > crawl_000_111/segment -numFetchers 2 -mapper=1 -reducer=1 -noFilter -noNorm > -force > {code} > 1st time it returns : > {code} > 2022-11-23 11:25:15,202 WARN crawl.Generator: Generator: 0 records selected > for fetching, exiting ... > {code} > 2nd time it returns : > {code} > 2022-11-23 11:27:43,007 INFO crawl.Generator: Generator: Partitioning > selected urls for politeness. > 2022-11-23 11:27:44,009 INFO crawl.Generator: Generator: segment: > crawl_000_111/segment/20221123112744 > ... > 2022-11-23 11:28:34,061 INFO crawl.Generator: Generator: finished at > 2022-11-23 11:28:34, elapsed: 00:01:53 > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)