[jira] [Created] (NUTCH-1562) Order of execution for scoring filters
Julien Nioche created NUTCH-1562:
Summary: Order of execution for scoring filters
Key: NUTCH-1562
URL: https://issues.apache.org/jira/browse/NUTCH-1562
Project: Nutch
Issue Type: Bug
Components: documentation
Affects Versions: 2.1, 1.6
Reporter: Julien Nioche
Fix For: 1.7, 2.2
The documentation in nutch-default.xml states that :
{quote}
property
namescoring.filter.order/name
value/value
descriptionThe order in which scoring filters are applied.
This may be left empty (in which case all available scoring
filters will be applied in the order defined in plugin-includes
and plugin-excludes), or a space separated list of implementation
classes.
/description
/property
{quote}
however if no order is specified the filters are ordered randomly and not in
the order defined in plugin-includes.
The other *order parameters (e.g. urlfilter.order) have a different
documentation and are loaded and applied in system defined order which
corresponds to what the code does.
The patch attached is for 1.x and puts the code in accordance with the
documentation by ordering the filters according to the order of the plugins,
which gives users more control without having to specify the classes explicitly
in scoring.filter.order.
We could extend the same idea to the other *order params.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1562) Order of execution for scoring filters
[
https://issues.apache.org/jira/browse/NUTCH-1562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julien Nioche updated NUTCH-1562:
-
Attachment: NUTCH-1562-trunk.patch
Patch for trunk
Order of execution for scoring filters
--
Key: NUTCH-1562
URL: https://issues.apache.org/jira/browse/NUTCH-1562
Project: Nutch
Issue Type: Bug
Components: documentation
Affects Versions: 1.6, 2.1
Reporter: Julien Nioche
Fix For: 1.7, 2.2
Attachments: NUTCH-1562-trunk.patch
The documentation in nutch-default.xml states that :
{quote}
property
namescoring.filter.order/name
value/value
descriptionThe order in which scoring filters are applied.
This may be left empty (in which case all available scoring
filters will be applied in the order defined in plugin-includes
and plugin-excludes), or a space separated list of implementation
classes.
/description
/property
{quote}
however if no order is specified the filters are ordered randomly and not in
the order defined in plugin-includes.
The other *order parameters (e.g. urlfilter.order) have a different
documentation and are loaded and applied in system defined order which
corresponds to what the code does.
The patch attached is for 1.x and puts the code in accordance with the
documentation by ordering the filters according to the order of the plugins,
which gives users more control without having to specify the classes
explicitly in scoring.filter.order.
We could extend the same idea to the other *order params.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Build failed in Jenkins: Nutch-nutchgora #574
See https://builds.apache.org/job/Nutch-nutchgora/574/changes Changes: [jnioche] Fixed detection of job file for distributed mode -- [...truncated 2916 lines...] clean-lib: resolve-default: [ivy:resolve] :: loading settings :: file = /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-nutchgora/2.x/ivy/ivysettings.xml compile: [echo] Compiling plugin: urlfilter-validator [javac] /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-nutchgora/2.x/src/plugin/build-plugin.xml:117: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds [javac] Compiling 1 source file to /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-nutchgora/2.x/build/urlfilter-validator/classes [javac] warning: [options] bootstrap class path not set in conjunction with -source 1.6 [javac] 1 warning jar: [jar] Building jar: /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-nutchgora/2.x/build/urlfilter-validator/urlfilter-validator.jar deps-test: deploy: [copy] Copying 1 file to /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-nutchgora/2.x/build/plugins/urlfilter-validator copy-generated-lib: [copy] Copying 1 file to /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-nutchgora/2.x/build/plugins/urlfilter-validator init: [mkdir] Created dir: /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-nutchgora/2.x/build/urlnormalizer-basic [mkdir] Created dir: /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-nutchgora/2.x/build/urlnormalizer-basic/classes [mkdir] Created dir: /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-nutchgora/2.x/build/urlnormalizer-basic/test [mkdir] Created dir: /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-nutchgora/2.x/build/plugins/urlnormalizer-basic init-plugin: deps-jar: clean-lib: resolve-default: [ivy:resolve] :: loading settings :: file = /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-nutchgora/2.x/ivy/ivysettings.xml compile: [echo] Compiling plugin: urlnormalizer-basic [javac] /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-nutchgora/2.x/src/plugin/build-plugin.xml:117: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds [javac] Compiling 1 source file to /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-nutchgora/2.x/build/urlnormalizer-basic/classes [javac] warning: [options] bootstrap class path not set in conjunction with -source 1.6 [javac] 1 warning jar: [jar] Building jar: /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-nutchgora/2.x/build/urlnormalizer-basic/urlnormalizer-basic.jar deps-test: deploy: [copy] Copying 1 file to /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-nutchgora/2.x/build/plugins/urlnormalizer-basic copy-generated-lib: [copy] Copying 1 file to /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-nutchgora/2.x/build/plugins/urlnormalizer-basic init: [mkdir] Created dir: /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-nutchgora/2.x/build/urlnormalizer-pass [mkdir] Created dir: /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-nutchgora/2.x/build/urlnormalizer-pass/classes [mkdir] Created dir: /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-nutchgora/2.x/build/urlnormalizer-pass/test [mkdir] Created dir: /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-nutchgora/2.x/build/plugins/urlnormalizer-pass init-plugin: deps-jar: clean-lib: resolve-default: [ivy:resolve] :: loading settings :: file = /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-nutchgora/2.x/ivy/ivysettings.xml compile: [echo] Compiling plugin: urlnormalizer-pass [javac] /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-nutchgora/2.x/src/plugin/build-plugin.xml:117: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds [javac] Compiling 1 source file to /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-nutchgora/2.x/build/urlnormalizer-pass/classes [javac] warning: [options] bootstrap class path not set in conjunction with -source 1.6 [javac] 1 warning jar: [jar] Building jar: /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-nutchgora/2.x/build/urlnormalizer-pass/urlnormalizer-pass.jar deps-test: deploy: [copy] Copying 1 file to /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-nutchgora/2.x/build/plugins/urlnormalizer-pass
Build failed in Jenkins: Nutch-trunk #2173
See https://builds.apache.org/job/Nutch-trunk/2173/changes
Changes:
[jnioche] Fixed detection of job file for distributed mode
--
[...truncated 4018 lines...]
[javac]
/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-trunk/trunk/src/test/org/apache/nutch/crawl/TestCrawlDbMerger.java:76:
warning: [unchecked] unchecked call to put(K,V) as a member of the raw type
HashMap
[javac] expected.put(url10, cd3);
[javac] ^
[javac] where K,V are type-variables:
[javac] K extends Object declared in class HashMap
[javac] V extends Object declared in class HashMap
[javac]
/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-trunk/trunk/src/test/org/apache/nutch/crawl/TestCrawlDbMerger.java:77:
warning: [unchecked] unchecked call to put(K,V) as a member of the raw type
HashMap
[javac] expected.put(url11, cd1);
[javac] ^
[javac] where K,V are type-variables:
[javac] K extends Object declared in class HashMap
[javac] V extends Object declared in class HashMap
[javac]
/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-trunk/trunk/src/test/org/apache/nutch/crawl/TestCrawlDbMerger.java:78:
warning: [unchecked] unchecked call to put(K,V) as a member of the raw type
HashMap
[javac] expected.put(url21, cd2);
[javac] ^
[javac] where K,V are type-variables:
[javac] K extends Object declared in class HashMap
[javac] V extends Object declared in class HashMap
[javac]
/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-trunk/trunk/src/test/org/apache/nutch/crawl/TestCrawlDbMerger.java:89:
warning: [deprecation] delete(Path) in FileSystem has been deprecated
[javac] fs.delete(testDir);
[javac] ^
[javac]
/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-trunk/trunk/src/test/org/apache/nutch/crawl/TestCrawlDbMerger.java:108:
warning: [rawtypes] found raw type: Iterator
[javac] Iterator it = expected.keySet().iterator();
[javac] ^
[javac] missing type arguments for generic class IteratorE
[javac] where E is a type-variable:
[javac] E extends Object declared in interface Iterator
[javac]
/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-trunk/trunk/src/test/org/apache/nutch/crawl/TestCrawlDbMerger.java:123:
warning: [deprecation] delete(Path) in FileSystem has been deprecated
[javac] fs.delete(testDir);
[javac] ^
[javac]
/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-trunk/trunk/src/test/org/apache/nutch/crawl/TestCrawlDbMerger.java:126:
warning: [rawtypes] found raw type: TreeSet
[javac] private void createCrawlDb(Configuration config, FileSystem fs,
Path crawldb, TreeSet init, CrawlDatum cd) throws Exception {
[javac]
^
[javac] missing type arguments for generic class TreeSetE
[javac] where E is a type-variable:
[javac] E extends Object declared in class TreeSet
[javac]
/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-trunk/trunk/src/test/org/apache/nutch/crawl/TestCrawlDbMerger.java:130:
warning: [rawtypes] found raw type: Iterator
[javac] Iterator it = init.iterator();
[javac] ^
[javac] missing type arguments for generic class IteratorE
[javac] where E is a type-variable:
[javac] E extends Object declared in interface Iterator
[javac]
/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-trunk/trunk/src/test/org/apache/nutch/crawl/TestLinkDbMerger.java:71:
warning: [rawtypes] found raw type: TreeMap
[javac] TreeMap init1 = new TreeMap();
[javac] ^
[javac] missing type arguments for generic class TreeMapK,V
[javac] where K,V are type-variables:
[javac] K extends Object declared in class TreeMap
[javac] V extends Object declared in class TreeMap
[javac]
/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-trunk/trunk/src/test/org/apache/nutch/crawl/TestLinkDbMerger.java:71:
warning: [rawtypes] found raw type: TreeMap
[javac] TreeMap init1 = new TreeMap();
[javac] ^
[javac] missing type arguments for generic class TreeMapK,V
[javac] where K,V are type-variables:
[javac] K extends Object declared in class TreeMap
[javac] V extends Object declared in class TreeMap
[javac]
/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-trunk/trunk/src/test/org/apache/nutch/crawl/TestLinkDbMerger.java:72:
warning: [rawtypes] found raw type: TreeMap
[javac] TreeMap init2 = new TreeMap();
[javac] ^
[javac] missing type arguments for generic class TreeMapK,V
Re: FetchSchedule and Metadata
yes , the method of FetchSchedule#getFields() is never used. so maybe we
GenerateJob, we can add an method to get the fields before initMapperJob.
code like this
GeneratorJob#getFields
public CollectionWebPage.Field getFields(Job job) {
CollectionWebPage.Field fields = new HashSetWebPage.Field(FIELDS);
fields.addAll(FetchScheduleFactory.getFetchSchedule(job.getConfiguration()).getFields(););
return fields;
}
GenerateorJob#run
currentJob = new NutchJob(getConf(), generate: +
getConf().get(BATCH_ID));
CollectionWebPage.Field fields = getFields(currentJob);
StorageUtils.initMapperJob(currentJob, fields, SelectorEntry.class,
WebPage.class, GeneratorMapper.class,
SelectorEntryPartitioner.class, true);
On Thu, Apr 18, 2013 at 1:56 AM, Lewis John Mcgibbney
lewis.mcgibb...@gmail.com wrote:
Hi Canan,
On Tue, Apr 16, 2013 at 9:46 PM, dev-digest-h...@nutch.apache.org wrote:
dev Digest 17 Apr 2013 04:46:54 - Issue 1595
Re: FetchSchedule and Metadata
23108 by: Canan GİRGİN
Hi,
Can I open an issue in Jira system about FetchSchedules getFields()
method?
This method is never used. But it is necessary.
I extend AbstractFetchSchedule class for my specific schedule
requirements. But I can not use required columns data in my class.(Ex:
metadata column)
Yes I would please encourage you to open an issue for this. I am still not
completely sure what the problem is (and no one else seems to have chimed
in here) so please explain your problem as verbosely as possible and we can
work on it.
Sorry for loosing track of this issue.
Lewis
--
Don't Grow Old, Grow Up... :-)
[jira] [Created] (NUTCH-1563) FetchSchedule#getFields is never used by GeneraterJob
lufeng created NUTCH-1563: - Summary: FetchSchedule#getFields is never used by GeneraterJob Key: NUTCH-1563 URL: https://issues.apache.org/jira/browse/NUTCH-1563 Project: Nutch Issue Type: Bug Components: generator Affects Versions: 2.1 Reporter: lufeng Priority: Minor Fix For: 2.2 The method of getFields in FetchSchedule if never used, so if user extends the FetchSchedule and want to get some fields of WebPage, it always return null. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: FetchSchedule and Metadata
Hi, Canan I have added a issue in Jira. https://issues.apache.org/jira/browse/NUTCH-1563 and attached a fixed patch in it. you can try it. thanks for your report. -- Don't Grow Old, Grow Up... :-)
Build failed in Jenkins: Nutch-trunk #2174
See https://builds.apache.org/job/Nutch-trunk/2174/
--
[...truncated 4045 lines...]
[javac] V extends Object declared in class HashMap
[javac]
/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-trunk/trunk/src/test/org/apache/nutch/crawl/TestCrawlDbMerger.java:89:
warning: [deprecation] delete(Path) in FileSystem has been deprecated
[javac] fs.delete(testDir);
[javac] ^
[javac]
/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-trunk/trunk/src/test/org/apache/nutch/crawl/TestCrawlDbMerger.java:108:
warning: [rawtypes] found raw type: Iterator
[javac] Iterator it = expected.keySet().iterator();
[javac] ^
[javac] missing type arguments for generic class IteratorE
[javac] where E is a type-variable:
[javac] E extends Object declared in interface Iterator
[javac]
/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-trunk/trunk/src/test/org/apache/nutch/crawl/TestCrawlDbMerger.java:123:
warning: [deprecation] delete(Path) in FileSystem has been deprecated
[javac] fs.delete(testDir);
[javac] ^
[javac]
/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-trunk/trunk/src/test/org/apache/nutch/crawl/TestCrawlDbMerger.java:126:
warning: [rawtypes] found raw type: TreeSet
[javac] private void createCrawlDb(Configuration config, FileSystem fs,
Path crawldb, TreeSet init, CrawlDatum cd) throws Exception {
[javac]
^
[javac] missing type arguments for generic class TreeSetE
[javac] where E is a type-variable:
[javac] E extends Object declared in class TreeSet
[javac]
/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-trunk/trunk/src/test/org/apache/nutch/crawl/TestCrawlDbMerger.java:130:
warning: [rawtypes] found raw type: Iterator
[javac] Iterator it = init.iterator();
[javac] ^
[javac] missing type arguments for generic class IteratorE
[javac] where E is a type-variable:
[javac] E extends Object declared in interface Iterator
[javac]
/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-trunk/trunk/src/test/org/apache/nutch/crawl/TestLinkDbMerger.java:71:
warning: [rawtypes] found raw type: TreeMap
[javac] TreeMap init1 = new TreeMap();
[javac] ^
[javac] missing type arguments for generic class TreeMapK,V
[javac] where K,V are type-variables:
[javac] K extends Object declared in class TreeMap
[javac] V extends Object declared in class TreeMap
[javac]
/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-trunk/trunk/src/test/org/apache/nutch/crawl/TestLinkDbMerger.java:71:
warning: [rawtypes] found raw type: TreeMap
[javac] TreeMap init1 = new TreeMap();
[javac] ^
[javac] missing type arguments for generic class TreeMapK,V
[javac] where K,V are type-variables:
[javac] K extends Object declared in class TreeMap
[javac] V extends Object declared in class TreeMap
[javac]
/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-trunk/trunk/src/test/org/apache/nutch/crawl/TestLinkDbMerger.java:72:
warning: [rawtypes] found raw type: TreeMap
[javac] TreeMap init2 = new TreeMap();
[javac] ^
[javac] missing type arguments for generic class TreeMapK,V
[javac] where K,V are type-variables:
[javac] K extends Object declared in class TreeMap
[javac] V extends Object declared in class TreeMap
[javac]
/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-trunk/trunk/src/test/org/apache/nutch/crawl/TestLinkDbMerger.java:72:
warning: [rawtypes] found raw type: TreeMap
[javac] TreeMap init2 = new TreeMap();
[javac] ^
[javac] missing type arguments for generic class TreeMapK,V
[javac] where K,V are type-variables:
[javac] K extends Object declared in class TreeMap
[javac] V extends Object declared in class TreeMap
[javac]
/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-trunk/trunk/src/test/org/apache/nutch/crawl/TestLinkDbMerger.java:73:
warning: [rawtypes] found raw type: HashMap
[javac] HashMap expected = new HashMap();
[javac] ^
[javac] missing type arguments for generic class HashMapK,V
[javac] where K,V are type-variables:
[javac] K extends Object declared in class HashMap
[javac] V extends Object declared in class HashMap
[javac]
/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-trunk/trunk/src/test/org/apache/nutch/crawl/TestLinkDbMerger.java:73:
warning: [rawtypes] found raw type: HashMap
[javac] HashMap expected = new HashMap();
[javac] ^
Jenkins build is back to normal : Nutch-nutchgora #575
See https://builds.apache.org/job/Nutch-nutchgora/575/
