[jira] [Created] (NUTCH-2248) CSS parser plugin
Joseph Naegele created NUTCH-2248: - Summary: CSS parser plugin Key: NUTCH-2248 URL: https://issues.apache.org/jira/browse/NUTCH-2248 Project: Nutch Issue Type: New Feature Components: parser, plugin Affects Versions: 1.12 Reporter: Joseph Naegele This plugin allows for collecting {{uri}} links from CSS (stylesheets). This is useful for collecting parent stylesheets, fonts, and images needed to display web pages as intended. Parsed Outlinks do not have associated anchors, and no additional text/content is parsed from the stylesheet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NUTCH-1687) Pick queue in Round Robin
[ https://issues.apache.org/jira/browse/NUTCH-1687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15298757#comment-15298757 ] Joseph Naegele commented on NUTCH-1687: --- Any issues with Tien's updated patch? > Pick queue in Round Robin > - > > Key: NUTCH-1687 > URL: https://issues.apache.org/jira/browse/NUTCH-1687 > Project: Nutch > Issue Type: Improvement > Components: fetcher >Reporter: Tien Nguyen Manh >Priority: Minor > Attachments: NUTCH-1687-2.patch, NUTCH-1687.patch, > NUTCH-1687.tejasp.v1.patch > > > Currently we chose queue to pick url from start of queues list, so queue at > the start of list have more change to be pick first, that can cause problem > of long tail queue, which only few queue available at the end which have many > urls. > public synchronized FetchItem getFetchItem() { > final Iterator> it = > queues.entrySet().iterator(); ==> always reset to find queue from > start > while (it.hasNext()) { > > I think it is better to pick queue in round robin, that can make reduce time > to find the available queue and make all queue was picked in round robin and > if we use TopN during generator there are no long tail queue at the end. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NUTCH-2234) Upgrade to elasticsearch 2.1.1
[ https://issues.apache.org/jira/browse/NUTCH-2234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15298791#comment-15298791 ] Joseph Naegele commented on NUTCH-2234: --- Since this also adds support for multiple, comma-separated Elasticsearch hosts in {{elastic.host}}, the description {{nutch-default.xml}} should be updated accordingly. Is there any reason not to update this to use the most recent version of Elasticsearch (2.3.3)? I can update the patch or open a PR on Github. > Upgrade to elasticsearch 2.1.1 > -- > > Key: NUTCH-2234 > URL: https://issues.apache.org/jira/browse/NUTCH-2234 > Project: Nutch > Issue Type: Improvement > Components: indexer >Affects Versions: 1.11 >Reporter: Tien Nguyen Manh >Assignee: Markus Jelsma > Fix For: 1.13 > > Attachments: NUTCH-2234.patch > > > Currently we use elasticsearch 1.x, We should upgrade to 2.x -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NUTCH-2234) Upgrade to elasticsearch 2.1.1
[ https://issues.apache.org/jira/browse/NUTCH-2234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15298924#comment-15298924 ] Joseph Naegele commented on NUTCH-2234: --- Hmm I'm a bit confused. ES 2.3.3 depends on Lucene 5.5.0 libraries. It appears indexer-solr does not depend on Lucene, only Solrj. lucene-analyzers-common 4.10.2 is a Nutch-wide dependency in ivy/ivy.xml, but it appears to only be used by plugins: indexer-elastic, parsefilter-naivebayes, and scoring-similarity, of which indexer-elastic and parsefilter-naivebayes specify their Lucene dependencies in their own plugin.xml (scoring-similarity appears to rely on lucene-core 4.10.2 being a transitive dependency through lucene-analyzers-common. Changing the lucene version in ivy/ivy.xml requires changes to the scoring-similarity plugin, which I think should be its own issue. > Upgrade to elasticsearch 2.1.1 > -- > > Key: NUTCH-2234 > URL: https://issues.apache.org/jira/browse/NUTCH-2234 > Project: Nutch > Issue Type: Improvement > Components: indexer >Affects Versions: 1.11 >Reporter: Tien Nguyen Manh >Assignee: Markus Jelsma > Fix For: 1.13 > > Attachments: NUTCH-2234.patch > > > Currently we use elasticsearch 1.x, We should upgrade to 2.x -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NUTCH-2234) Upgrade to elasticsearch 2.1.1
[ https://issues.apache.org/jira/browse/NUTCH-2234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15300590#comment-15300590 ] Joseph Naegele commented on NUTCH-2234: --- Understood. The update to Lucene analyzers requires minor programmatic API changes in scoring-similarity, but nothing big. None of the indexers have tests, so I'm testing indexer-elastic manually for now. Unfortunately updating Elasticsearch breaks the plugin due to differences in guava versions: indexer-elastic depends on guava-18.0, which it declares in its plugin.xml, but guava-16.0.1 is a Nutch-wide dependency (for Hadoop). We avoided this issue in the past by also updating Nutch's Hadoop dependency from 2.4.0 -> 2.7.1, which is why Tien created NUTCH-2246. I'll open the PR with all aforementioned dependency updates. > Upgrade to elasticsearch 2.1.1 > -- > > Key: NUTCH-2234 > URL: https://issues.apache.org/jira/browse/NUTCH-2234 > Project: Nutch > Issue Type: Improvement > Components: indexer >Affects Versions: 1.11 >Reporter: Tien Nguyen Manh >Assignee: Markus Jelsma > Fix For: 1.13 > > Attachments: NUTCH-2234.patch > > > Currently we use elasticsearch 1.x, We should upgrade to 2.x -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NUTCH-1611) Elastic Search Indexer Creates field in elastic search "boost" as a string value, so cannot be used in custom boost queries
[ https://issues.apache.org/jira/browse/NUTCH-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Naegele updated NUTCH-1611: -- Attachment: NUTCH-1611-1.x.patch This is an issue in 1.x. I've attached a patch for 1.x for which all tests pass. > Elastic Search Indexer Creates field in elastic search "boost" as a string > value, so cannot be used in custom boost queries > --- > > Key: NUTCH-1611 > URL: https://issues.apache.org/jira/browse/NUTCH-1611 > Project: Nutch > Issue Type: Bug > Components: indexer >Affects Versions: 2.2.1 > Environment: All >Reporter: Nicholas Waltham > Attachments: NUTCH-1611-1.x.patch > > > Ordinarily, one can use a boost field in a custom_score query in elastic > search to affect the ranking, nutch create such a field. However it is store > in elastic search as a string, so cannot be used. Attempt to use the boost > field in a query therefore creates the following error: > PropertyAccessException[[Error: could not access: floatValue; in class: > org.elasticsearch.index.field.data.strings.StringDocFieldData]\n[Near : {... > _score + (1 * doc.boost.floatValue / 100) }] > example test query: > { > "query" : { > "custom_score" : { > "query" : { > "query_string" : { > "query" : "something" > }}, > "script" : "_score + (1 * doc.boost.doubleValue / 100)" > } >} > } -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NUTCH-2279) LinkRank fails when using Hadoop MR output compression
Joseph Naegele created NUTCH-2279: - Summary: LinkRank fails when using Hadoop MR output compression Key: NUTCH-2279 URL: https://issues.apache.org/jira/browse/NUTCH-2279 Project: Nutch Issue Type: Bug Affects Versions: 1.11 Reporter: Joseph Naegele When using MapReduce job output compression, i.e. {{mapreduce.output.fileoutputformat.compress=true}}, LinkRank can't read the results of its {{Counter}} MR job due to the additional, generated file extension. For example, using the default compression codec (which appears to be DEFLATE), the counter file is written to {{crawl/webgraph/_num_nodes_/part-0.deflate}}. Then, the LinkRank job attempts to manually read this file to obtain the number of links using the following code: {code} FSDataInputStream readLinks = fs.open(new Path(numLinksPath, "part-0")); {code} which fails because the file {{part-0}} doesn't exist: {code} LinkAnalysis: java.io.FileNotFoundException: File crawl/webgraph/_num_nodes_/part-0 does not exist at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:606) at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:819) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:596) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421) at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:140) at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:341) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767) at org.apache.nutch.scoring.webgraph.LinkRank.runCounter(LinkRank.java:124) at org.apache.nutch.scoring.webgraph.LinkRank.analyze(LinkRank.java:633) at org.apache.nutch.scoring.webgraph.LinkRank.run(LinkRank.java:713) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.nutch.scoring.webgraph.LinkRank.main(LinkRank.java:680) {code} To reproduce, add {{-D mapreduce.output.fileoutputformat.compress=true}} to the properties for {{bin/nutch linkrank ...}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NUTCH-2279) LinkRank fails when using Hadoop MR output compression
[ https://issues.apache.org/jira/browse/NUTCH-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Naegele updated NUTCH-2279: -- Affects Version/s: (was: 1.11) 1.12 > LinkRank fails when using Hadoop MR output compression > -- > > Key: NUTCH-2279 > URL: https://issues.apache.org/jira/browse/NUTCH-2279 > Project: Nutch > Issue Type: Bug >Affects Versions: 1.12 >Reporter: Joseph Naegele > > When using MapReduce job output compression, i.e. > {{mapreduce.output.fileoutputformat.compress=true}}, LinkRank can't read the > results of its {{Counter}} MR job due to the additional, generated file > extension. > For example, using the default compression codec (which appears to be > DEFLATE), the counter file is written to > {{crawl/webgraph/_num_nodes_/part-0.deflate}}. Then, the LinkRank job > attempts to manually read this file to obtain the number of links using the > following code: > {code} > FSDataInputStream readLinks = fs.open(new Path(numLinksPath, "part-0")); > {code} > which fails because the file {{part-0}} doesn't exist: > {code} > LinkAnalysis: java.io.FileNotFoundException: File > crawl/webgraph/_num_nodes_/part-0 does not exist > at > org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:606) > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:819) > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:596) > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421) > at > org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:140) > at > org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:341) > at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767) > at > org.apache.nutch.scoring.webgraph.LinkRank.runCounter(LinkRank.java:124) > at > org.apache.nutch.scoring.webgraph.LinkRank.analyze(LinkRank.java:633) > at org.apache.nutch.scoring.webgraph.LinkRank.run(LinkRank.java:713) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.nutch.scoring.webgraph.LinkRank.main(LinkRank.java:680) > {code} > To reproduce, add {{-D mapreduce.output.fileoutputformat.compress=true}} to > the properties for {{bin/nutch linkrank ...}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NUTCH-2287) Indexer-elastic plugin should use Elasticsearch BulkProcessor and BackoffPolicy
Joseph Naegele created NUTCH-2287: - Summary: Indexer-elastic plugin should use Elasticsearch BulkProcessor and BackoffPolicy Key: NUTCH-2287 URL: https://issues.apache.org/jira/browse/NUTCH-2287 Project: Nutch Issue Type: Improvement Components: indexer, plugin Affects Versions: 1.12 Reporter: Joseph Naegele Elasticsearch's API (since at least v2.0) includes the {{BulkProcessor}}, which automatically handles flushing bulk requests given a max doc count and/or max bulk size. It also now (I believe since 2.2.0) offers a {{BackoffPolicy}} option, allowing the BulkProcessor/Client to retry bulk requests when the Elasticsearch cluster is saturated. Using the {{BulkProcessor}} was originally suggested [here|https://issues.apache.org/jira/browse/NUTCH-1527?focusedCommentId=1316&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-1316]. Refactoring the {{indexer-elastic}} plugin to use the {{BulkProcessor}} will greatly simplify the existing plugin at the cost of slightly less debug logging. Additionally, it will allow the plugin to handle cluster saturation gracefully (rather than raising a RuntimeException and killing the reduce task), by using a configurable "exponential back-off policy". https://www.elastic.co/guide/en/elasticsearch/client/java-api/2.3/java-docs-bulk-processor.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NUTCH-2287) Indexer-elastic plugin should use Elasticsearch BulkProcessor and BackoffPolicy
[ https://issues.apache.org/jira/browse/NUTCH-2287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15348659#comment-15348659 ] Joseph Naegele commented on NUTCH-2287: --- I can have a PR ready early next week. This will require upgrading the Elasticsearch dependency to at least version 2.2.0 (potentially handled by NUTCH-2234, which upgrades it to 2.3.3). > Indexer-elastic plugin should use Elasticsearch BulkProcessor and > BackoffPolicy > --- > > Key: NUTCH-2287 > URL: https://issues.apache.org/jira/browse/NUTCH-2287 > Project: Nutch > Issue Type: Improvement > Components: indexer, plugin >Affects Versions: 1.12 >Reporter: Joseph Naegele > > Elasticsearch's API (since at least v2.0) includes the {{BulkProcessor}}, > which automatically handles flushing bulk requests given a max doc count > and/or max bulk size. It also now (I believe since 2.2.0) offers a > {{BackoffPolicy}} option, allowing the BulkProcessor/Client to retry bulk > requests when the Elasticsearch cluster is saturated. Using the > {{BulkProcessor}} was originally suggested > [here|https://issues.apache.org/jira/browse/NUTCH-1527?focusedCommentId=1316&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-1316]. > Refactoring the {{indexer-elastic}} plugin to use the {{BulkProcessor}} will > greatly simplify the existing plugin at the cost of slightly less debug > logging. Additionally, it will allow the plugin to handle cluster saturation > gracefully (rather than raising a RuntimeException and killing the reduce > task), by using a configurable "exponential back-off policy". > https://www.elastic.co/guide/en/elasticsearch/client/java-api/2.3/java-docs-bulk-processor.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)