[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2016-03-19 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15203025#comment-15203025
 ] 

ASF GitHub Bot commented on NUTCH-961:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nutch/pull/92


> Expose Tika's boilerpipe support
> 
>
> Key: NUTCH-961
> URL: https://issues.apache.org/jira/browse/NUTCH-961
> Project: Nutch
>  Issue Type: New Feature
>  Components: parser
>Affects Versions: 1.11
>Reporter: Markus Jelsma
>Assignee: Markus Jelsma
> Fix For: 1.12
>
> Attachments: BoilerpipeExtractorRepository.java, 
> NUTCH-961-1.11-1.patch, NUTCH-961-1.3-3.patch, 
> NUTCH-961-1.3-tikaparser.patch, NUTCH-961-1.3-tikaparser1.patch, 
> NUTCH-961-1.4-dombuilder-1.patch, NUTCH-961-1.5-1.patch, 
> NUTCH-961-1.8-1.patch, NUTCH-961-2.1-v1.patch, NUTCH-961-2.1-v2.patch, 
> NUTCH-961.patch, NUTCH-961.patch, NUTCH-961v2.patch, 
> nutch-2.x-boilerpipe.patch
>
>
> Tika 0.8 comes with the Boilerpipe content handler which can be used to 
> extract boilerplate content from HTML pages. We should see how we can expose 
> Boilerplate in the Nutch cofiguration.
> Use the following properties to enable and control Boilerpipe.
> {code}
> 
>   tika.extractor
>   none
>   
>   Which text extraction algorithm to use. Valid values are: boilerpipe or 
> none.
>   
> 
>  
>  
>   tika.extractor.boilerpipe.algorithm
>   ArticleExtractor
>
>   Which Boilerpipe algorithm to use. Valid values are: DefaultExtractor, 
> ArticleExtractor
>   or CanolaExtractor.
>   
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2016-02-27 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15170557#comment-15170557
 ] 

ASF GitHub Bot commented on NUTCH-961:
--

Github user lewismc commented on a diff in the pull request:

https://github.com/apache/nutch/pull/92#discussion_r54332201
  
--- Diff: 
src/plugin/parse-tika/src/java/org/apache/nutch/parse/tika/TikaParser.java ---
@@ -109,7 +114,18 @@ public Parse getParse(String url, WebPage page) {
 HTMLDocumentImpl doc = new HTMLDocumentImpl();
 doc.setErrorChecking(false);
 DocumentFragment root = doc.createDocumentFragment();
-DOMBuilder domhandler = new DOMBuilder(doc, root);
+   // DOMBuilder domhandler = new DOMBuilder(doc, root);
+ContentHandler domHandler;
+// Check whether to use Tika's BoilerplateContentHandler
+if (useBoilerpipe) {
+LOG.debug("Using Tikas's Boilerpipe with Extractor: " + 
boilerpipeExtractorName);
--- End diff --

Can also use more efficient slf4j convention
logger.debug("The entry is {}.", entry);


> Expose Tika's boilerpipe support
> 
>
> Key: NUTCH-961
> URL: https://issues.apache.org/jira/browse/NUTCH-961
> Project: Nutch
>  Issue Type: New Feature
>  Components: parser
>Affects Versions: 1.11
>Reporter: Markus Jelsma
>Assignee: Markus Jelsma
> Fix For: 1.12
>
> Attachments: BoilerpipeExtractorRepository.java, 
> NUTCH-961-1.11-1.patch, NUTCH-961-1.3-3.patch, 
> NUTCH-961-1.3-tikaparser.patch, NUTCH-961-1.3-tikaparser1.patch, 
> NUTCH-961-1.4-dombuilder-1.patch, NUTCH-961-1.5-1.patch, 
> NUTCH-961-1.8-1.patch, NUTCH-961-2.1-v1.patch, NUTCH-961-2.1-v2.patch, 
> NUTCH-961.patch, NUTCH-961.patch, NUTCH-961v2.patch, 
> nutch-2.x-boilerpipe.patch
>
>
> Tika 0.8 comes with the Boilerpipe content handler which can be used to 
> extract boilerplate content from HTML pages. We should see how we can expose 
> Boilerplate in the Nutch cofiguration.
> Use the following properties to enable and control Boilerpipe.
> {code}
> 
>   tika.extractor
>   none
>   
>   Which text extraction algorithm to use. Valid values are: boilerpipe or 
> none.
>   
> 
>  
>  
>   tika.extractor.boilerpipe.algorithm
>   ArticleExtractor
>
>   Which Boilerpipe algorithm to use. Valid values are: DefaultExtractor, 
> ArticleExtractor
>   or CanolaExtractor.
>   
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2016-02-27 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15170556#comment-15170556
 ] 

ASF GitHub Bot commented on NUTCH-961:
--

Github user lewismc commented on a diff in the pull request:

https://github.com/apache/nutch/pull/92#discussion_r54332193
  
--- Diff: 
src/plugin/parse-tika/src/java/org/apache/nutch/parse/tika/BoilerpipeExtractorRepository.java
 ---
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nutch.parse.tika;
+
+import java.lang.ClassLoader;
+import java.lang.InstantiationException;
+import java.util.WeakHashMap;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.tika.parser.html.BoilerpipeContentHandler;
+import de.l3s.boilerpipe.BoilerpipeExtractor;
+import de.l3s.boilerpipe.extractors.*;
+
+class BoilerpipeExtractorRepository {
+
+public static final Log LOG = 
LogFactory.getLog(BoilerpipeExtractorRepository.class);
+public static final WeakHashMap 
extractorRepository = new WeakHashMap();
+ 
+/**
+ * Returns an instance of the specified extractor
+ */
+public static BoilerpipeExtractor getExtractor(String 
boilerpipeExtractorName) {
+  // Check if there's no instance of this extractor
+  if (!extractorRepository.containsKey(boilerpipeExtractorName)) {
+// FQCN
+boilerpipeExtractorName = "de.l3s.boilerpipe.extractors." + 
boilerpipeExtractorName;
+
+// Attempt to load the class
+try {
+  ClassLoader loader = BoilerpipeExtractor.class.getClassLoader();
+  Class extractorClass = loader.loadClass(boilerpipeExtractorName);
+
+  // Add an instance to the repository
+  extractorRepository.put(boilerpipeExtractorName, 
(BoilerpipeExtractor)extractorClass.newInstance());
+
+} catch (ClassNotFoundException e) {
+  LOG.error("BoilerpipeExtractor " + boilerpipeExtractorName + " 
not found!");
--- End diff --

In slf4j we can better structure the catch
http://www.slf4j.org/faq.html#logging_performance
e.g.
```
logger.debug("The entry is {}.", entry);
```


> Expose Tika's boilerpipe support
> 
>
> Key: NUTCH-961
> URL: https://issues.apache.org/jira/browse/NUTCH-961
> Project: Nutch
>  Issue Type: New Feature
>  Components: parser
>Affects Versions: 1.11
>Reporter: Markus Jelsma
>Assignee: Markus Jelsma
> Fix For: 1.12
>
> Attachments: BoilerpipeExtractorRepository.java, 
> NUTCH-961-1.11-1.patch, NUTCH-961-1.3-3.patch, 
> NUTCH-961-1.3-tikaparser.patch, NUTCH-961-1.3-tikaparser1.patch, 
> NUTCH-961-1.4-dombuilder-1.patch, NUTCH-961-1.5-1.patch, 
> NUTCH-961-1.8-1.patch, NUTCH-961-2.1-v1.patch, NUTCH-961-2.1-v2.patch, 
> NUTCH-961.patch, NUTCH-961.patch, NUTCH-961v2.patch, 
> nutch-2.x-boilerpipe.patch
>
>
> Tika 0.8 comes with the Boilerpipe content handler which can be used to 
> extract boilerplate content from HTML pages. We should see how we can expose 
> Boilerplate in the Nutch cofiguration.
> Use the following properties to enable and control Boilerpipe.
> {code}
> 
>   tika.extractor
>   none
>   
>   Which text extraction algorithm to use. Valid values are: boilerpipe or 
> none.
>   
> 
>  
>  
>   tika.extractor.boilerpipe.algorithm
>   ArticleExtractor
>
>   Which Boilerpipe algorithm to use. Valid values are: DefaultExtractor, 
> ArticleExtractor
>   or CanolaExtractor.
>   
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2016-02-27 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15170555#comment-15170555
 ] 

ASF GitHub Bot commented on NUTCH-961:
--

Github user lewismc commented on a diff in the pull request:

https://github.com/apache/nutch/pull/92#discussion_r54332155
  
--- Diff: 
src/plugin/parse-tika/src/java/org/apache/nutch/parse/tika/BoilerpipeExtractorRepository.java
 ---
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nutch.parse.tika;
+
+import java.lang.ClassLoader;
+import java.lang.InstantiationException;
+import java.util.WeakHashMap;
+import org.apache.commons.logging.Log;
--- End diff --

Nutch currently uses Slf4j

org.slf4j.Logger
org.slf4j.LoggerFactory

I think!


> Expose Tika's boilerpipe support
> 
>
> Key: NUTCH-961
> URL: https://issues.apache.org/jira/browse/NUTCH-961
> Project: Nutch
>  Issue Type: New Feature
>  Components: parser
>Affects Versions: 1.11
>Reporter: Markus Jelsma
>Assignee: Markus Jelsma
> Fix For: 1.12
>
> Attachments: BoilerpipeExtractorRepository.java, 
> NUTCH-961-1.11-1.patch, NUTCH-961-1.3-3.patch, 
> NUTCH-961-1.3-tikaparser.patch, NUTCH-961-1.3-tikaparser1.patch, 
> NUTCH-961-1.4-dombuilder-1.patch, NUTCH-961-1.5-1.patch, 
> NUTCH-961-1.8-1.patch, NUTCH-961-2.1-v1.patch, NUTCH-961-2.1-v2.patch, 
> NUTCH-961.patch, NUTCH-961.patch, NUTCH-961v2.patch, 
> nutch-2.x-boilerpipe.patch
>
>
> Tika 0.8 comes with the Boilerpipe content handler which can be used to 
> extract boilerplate content from HTML pages. We should see how we can expose 
> Boilerplate in the Nutch cofiguration.
> Use the following properties to enable and control Boilerpipe.
> {code}
> 
>   tika.extractor
>   none
>   
>   Which text extraction algorithm to use. Valid values are: boilerpipe or 
> none.
>   
> 
>  
>  
>   tika.extractor.boilerpipe.algorithm
>   ArticleExtractor
>
>   Which Boilerpipe algorithm to use. Valid values are: DefaultExtractor, 
> ArticleExtractor
>   or CanolaExtractor.
>   
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2016-02-27 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15170554#comment-15170554
 ] 

ASF GitHub Bot commented on NUTCH-961:
--

Github user lewismc commented on a diff in the pull request:

https://github.com/apache/nutch/pull/92#discussion_r54332145
  
--- Diff: conf/nutch-default.xml ---
@@ -876,6 +876,19 @@
   
 
 
+
+
+
+  tika.boilerpipe
+  false
--- End diff --

Can you provide descriptions of these properties please?


> Expose Tika's boilerpipe support
> 
>
> Key: NUTCH-961
> URL: https://issues.apache.org/jira/browse/NUTCH-961
> Project: Nutch
>  Issue Type: New Feature
>  Components: parser
>Affects Versions: 1.11
>Reporter: Markus Jelsma
>Assignee: Markus Jelsma
> Fix For: 1.12
>
> Attachments: BoilerpipeExtractorRepository.java, 
> NUTCH-961-1.11-1.patch, NUTCH-961-1.3-3.patch, 
> NUTCH-961-1.3-tikaparser.patch, NUTCH-961-1.3-tikaparser1.patch, 
> NUTCH-961-1.4-dombuilder-1.patch, NUTCH-961-1.5-1.patch, 
> NUTCH-961-1.8-1.patch, NUTCH-961-2.1-v1.patch, NUTCH-961-2.1-v2.patch, 
> NUTCH-961.patch, NUTCH-961.patch, NUTCH-961v2.patch, 
> nutch-2.x-boilerpipe.patch
>
>
> Tika 0.8 comes with the Boilerpipe content handler which can be used to 
> extract boilerplate content from HTML pages. We should see how we can expose 
> Boilerplate in the Nutch cofiguration.
> Use the following properties to enable and control Boilerpipe.
> {code}
> 
>   tika.extractor
>   none
>   
>   Which text extraction algorithm to use. Valid values are: boilerpipe or 
> none.
>   
> 
>  
>  
>   tika.extractor.boilerpipe.algorithm
>   ArticleExtractor
>
>   Which Boilerpipe algorithm to use. Valid values are: DefaultExtractor, 
> ArticleExtractor
>   or CanolaExtractor.
>   
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2016-02-26 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15168821#comment-15168821
 ] 

ASF GitHub Bot commented on NUTCH-961:
--

GitHub user jeremie70 opened a pull request:

https://github.com/apache/nutch/pull/92

Add the boilerpipe parsing adapted from NUTCH-961



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jeremie70/nutch my-branch

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nutch/pull/92.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #92


commit f185bc4461c57a1a85578de0ecf0884c7026c3a6
Author: Jérémie Bourseau 
Date:   2016-02-26T10:37:28Z

improve parser with boilerpipe

commit 93ea2e51f47be41ec93b2c0b0b61c117eeb3
Author: Jérémie Bourseau 
Date:   2016-02-26T10:37:28Z

NUTCH-961 improve parser with boilerpipe

commit be91764fdf59d4f6930fc3211a84a252e5452674
Author: Jérémie Bourseau 
Date:   2016-02-26T11:00:36Z

Merge branch 'my-branch' of https://github.com/jeremie70/nutch into 
my-branch




> Expose Tika's boilerpipe support
> 
>
> Key: NUTCH-961
> URL: https://issues.apache.org/jira/browse/NUTCH-961
> Project: Nutch
>  Issue Type: New Feature
>  Components: parser
>Affects Versions: 1.11
>Reporter: Markus Jelsma
>Assignee: Markus Jelsma
> Fix For: 1.12
>
> Attachments: BoilerpipeExtractorRepository.java, 
> NUTCH-961-1.11-1.patch, NUTCH-961-1.3-3.patch, 
> NUTCH-961-1.3-tikaparser.patch, NUTCH-961-1.3-tikaparser1.patch, 
> NUTCH-961-1.4-dombuilder-1.patch, NUTCH-961-1.5-1.patch, 
> NUTCH-961-1.8-1.patch, NUTCH-961-2.1-v1.patch, NUTCH-961-2.1-v2.patch, 
> NUTCH-961.patch, NUTCH-961.patch, NUTCH-961v2.patch, 
> nutch-2.x-boilerpipe.patch
>
>
> Tika 0.8 comes with the Boilerpipe content handler which can be used to 
> extract boilerplate content from HTML pages. We should see how we can expose 
> Boilerplate in the Nutch cofiguration.
> Use the following properties to enable and control Boilerpipe.
> {code}
> 
>   tika.extractor
>   none
>   
>   Which text extraction algorithm to use. Valid values are: boilerpipe or 
> none.
>   
> 
>  
>  
>   tika.extractor.boilerpipe.algorithm
>   ArticleExtractor
>
>   Which Boilerpipe algorithm to use. Valid values are: DefaultExtractor, 
> ArticleExtractor
>   or CanolaExtractor.
>   
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2016-02-16 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15148758#comment-15148758
 ] 

Hudson commented on NUTCH-961:
--

SUCCESS: Integrated in Nutch-trunk #3347 (See 
[https://builds.apache.org/job/Nutch-trunk/3347/])
NUTCH-961 Expose Tika's Boilerpipe support (markus: 
[http://svn.apache.org/viewvc/nutch/trunk/?view=rev=1730694])
* trunk/CHANGES.txt
* trunk/conf/nutch-default.xml
* 
trunk/src/plugin/parse-tika/src/java/org/apache/nutch/parse/tika/BoilerpipeExtractorRepository.java
* 
trunk/src/plugin/parse-tika/src/java/org/apache/nutch/parse/tika/TikaParser.java


> Expose Tika's boilerpipe support
> 
>
> Key: NUTCH-961
> URL: https://issues.apache.org/jira/browse/NUTCH-961
> Project: Nutch
>  Issue Type: New Feature
>  Components: parser
>Affects Versions: 1.11
>Reporter: Markus Jelsma
>Assignee: Markus Jelsma
> Fix For: 1.12
>
> Attachments: BoilerpipeExtractorRepository.java, 
> NUTCH-961-1.11-1.patch, NUTCH-961-1.3-3.patch, 
> NUTCH-961-1.3-tikaparser.patch, NUTCH-961-1.3-tikaparser1.patch, 
> NUTCH-961-1.4-dombuilder-1.patch, NUTCH-961-1.5-1.patch, 
> NUTCH-961-1.8-1.patch, NUTCH-961-2.1-v1.patch, NUTCH-961-2.1-v2.patch, 
> NUTCH-961.patch, NUTCH-961.patch, NUTCH-961v2.patch, 
> nutch-2.x-boilerpipe.patch
>
>
> Tika 0.8 comes with the Boilerpipe content handler which can be used to 
> extract boilerplate content from HTML pages. We should see how we can expose 
> Boilerplate in the Nutch cofiguration.
> Use the following properties to enable and control Boilerpipe.
> {code}
> 
>   tika.extractor
>   none
>   
>   Which text extraction algorithm to use. Valid values are: boilerpipe or 
> none.
>   
> 
>  
>  
>   tika.extractor.boilerpipe.algorithm
>   ArticleExtractor
>
>   Which Boilerpipe algorithm to use. Valid values are: DefaultExtractor, 
> ArticleExtractor
>   or CanolaExtractor.
>   
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2016-02-16 Thread Markus Jelsma (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15148642#comment-15148642
 ] 

Markus Jelsma commented on NUTCH-961:
-

Tests pass as expected and Boilerpipe as well. Will commit shortly.

> Expose Tika's boilerpipe support
> 
>
> Key: NUTCH-961
> URL: https://issues.apache.org/jira/browse/NUTCH-961
> Project: Nutch
>  Issue Type: New Feature
>  Components: parser
>Reporter: Markus Jelsma
>Assignee: Markus Jelsma
> Attachments: BoilerpipeExtractorRepository.java, 
> NUTCH-961-1.11-1.patch, NUTCH-961-1.3-3.patch, 
> NUTCH-961-1.3-tikaparser.patch, NUTCH-961-1.3-tikaparser1.patch, 
> NUTCH-961-1.4-dombuilder-1.patch, NUTCH-961-1.5-1.patch, 
> NUTCH-961-1.8-1.patch, NUTCH-961-2.1-v1.patch, NUTCH-961-2.1-v2.patch, 
> NUTCH-961.patch, NUTCH-961v2.patch, nutch-2.x-boilerpipe.patch
>
>
> Tika 0.8 comes with the Boilerpipe content handler which can be used to 
> extract boilerplate content from HTML pages. We should see how we can expose 
> Boilerplate in the Nutch cofiguration.
> Use the following properties to enable and control Boilerpipe.
> {code}
> 
>   tika.extractor
>   none
>   
>   Which text extraction algorithm to use. Valid values are: boilerpipe or 
> none.
>   
> 
>  
>  
>   tika.extractor.boilerpipe.algorithm
>   ArticleExtractor
>
>   Which Boilerpipe algorithm to use. Valid values are: DefaultExtractor, 
> ArticleExtractor
>   or CanolaExtractor.
>   
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2016-01-26 Thread Markus Jelsma (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15117024#comment-15117024
 ] 

Markus Jelsma commented on NUTCH-961:
-

Yes! :)

> Expose Tika's boilerpipe support
> 
>
> Key: NUTCH-961
> URL: https://issues.apache.org/jira/browse/NUTCH-961
> Project: Nutch
>  Issue Type: New Feature
>  Components: parser
>Reporter: Markus Jelsma
>Assignee: Markus Jelsma
> Attachments: BoilerpipeExtractorRepository.java, 
> NUTCH-961-1.11-1.patch, NUTCH-961-1.3-3.patch, 
> NUTCH-961-1.3-tikaparser.patch, NUTCH-961-1.3-tikaparser1.patch, 
> NUTCH-961-1.4-dombuilder-1.patch, NUTCH-961-1.5-1.patch, 
> NUTCH-961-1.8-1.patch, NUTCH-961-2.1-v1.patch, NUTCH-961-2.1-v2.patch, 
> NUTCH-961v2.patch, nutch-2.x-boilerpipe.patch
>
>
> Tika 0.8 comes with the Boilerpipe content handler which can be used to 
> extract boilerplate content from HTML pages. We should see how we can expose 
> Boilerplate in the Nutch cofiguration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2016-01-26 Thread Tien Nguyen Manh (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15117020#comment-15117020
 ] 

Tien Nguyen Manh commented on NUTCH-961:


Can NUTCH-1233: use tika to extract outlink solve that problem?

> Expose Tika's boilerpipe support
> 
>
> Key: NUTCH-961
> URL: https://issues.apache.org/jira/browse/NUTCH-961
> Project: Nutch
>  Issue Type: New Feature
>  Components: parser
>Reporter: Markus Jelsma
>Assignee: Markus Jelsma
> Attachments: BoilerpipeExtractorRepository.java, 
> NUTCH-961-1.11-1.patch, NUTCH-961-1.3-3.patch, 
> NUTCH-961-1.3-tikaparser.patch, NUTCH-961-1.3-tikaparser1.patch, 
> NUTCH-961-1.4-dombuilder-1.patch, NUTCH-961-1.5-1.patch, 
> NUTCH-961-1.8-1.patch, NUTCH-961-2.1-v1.patch, NUTCH-961-2.1-v2.patch, 
> NUTCH-961v2.patch, nutch-2.x-boilerpipe.patch
>
>
> Tika 0.8 comes with the Boilerpipe content handler which can be used to 
> extract boilerplate content from HTML pages. We should see how we can expose 
> Boilerplate in the Nutch cofiguration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2016-01-26 Thread Markus Jelsma (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15116975#comment-15116975
 ] 

Markus Jelsma commented on NUTCH-961:
-

With boilerpipe, you get only a very few outlinks, those found in the extracted 
text, and that is a problem :)

> Expose Tika's boilerpipe support
> 
>
> Key: NUTCH-961
> URL: https://issues.apache.org/jira/browse/NUTCH-961
> Project: Nutch
>  Issue Type: New Feature
>  Components: parser
>Reporter: Markus Jelsma
>Assignee: Markus Jelsma
> Attachments: BoilerpipeExtractorRepository.java, 
> NUTCH-961-1.11-1.patch, NUTCH-961-1.3-3.patch, 
> NUTCH-961-1.3-tikaparser.patch, NUTCH-961-1.3-tikaparser1.patch, 
> NUTCH-961-1.4-dombuilder-1.patch, NUTCH-961-1.5-1.patch, 
> NUTCH-961-1.8-1.patch, NUTCH-961-2.1-v1.patch, NUTCH-961-2.1-v2.patch, 
> NUTCH-961v2.patch, nutch-2.x-boilerpipe.patch
>
>
> Tika 0.8 comes with the Boilerpipe content handler which can be used to 
> extract boilerplate content from HTML pages. We should see how we can expose 
> Boilerplate in the Nutch cofiguration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2016-01-25 Thread Tien Nguyen Manh (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15116772#comment-15116772
 ] 

Tien Nguyen Manh commented on NUTCH-961:


AH yes, Could you explain why we need to parse it twice?

> Expose Tika's boilerpipe support
> 
>
> Key: NUTCH-961
> URL: https://issues.apache.org/jira/browse/NUTCH-961
> Project: Nutch
>  Issue Type: New Feature
>  Components: parser
>Reporter: Markus Jelsma
>Assignee: Markus Jelsma
> Attachments: BoilerpipeExtractorRepository.java, 
> NUTCH-961-1.11-1.patch, NUTCH-961-1.3-3.patch, 
> NUTCH-961-1.3-tikaparser.patch, NUTCH-961-1.3-tikaparser1.patch, 
> NUTCH-961-1.4-dombuilder-1.patch, NUTCH-961-1.5-1.patch, 
> NUTCH-961-1.8-1.patch, NUTCH-961-2.1-v1.patch, NUTCH-961-2.1-v2.patch, 
> NUTCH-961v2.patch, nutch-2.x-boilerpipe.patch
>
>
> Tika 0.8 comes with the Boilerpipe content handler which can be used to 
> extract boilerplate content from HTML pages. We should see how we can expose 
> Boilerplate in the Nutch cofiguration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2016-01-25 Thread Markus Jelsma (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15114989#comment-15114989
 ] 

Markus Jelsma commented on NUTCH-961:
-

That is probably due to the patch parsing twice. Once with BP for text, and 
once without for link extraction. 

> Expose Tika's boilerpipe support
> 
>
> Key: NUTCH-961
> URL: https://issues.apache.org/jira/browse/NUTCH-961
> Project: Nutch
>  Issue Type: New Feature
>  Components: parser
>Reporter: Markus Jelsma
>Assignee: Markus Jelsma
> Attachments: BoilerpipeExtractorRepository.java, 
> NUTCH-961-1.11-1.patch, NUTCH-961-1.3-3.patch, 
> NUTCH-961-1.3-tikaparser.patch, NUTCH-961-1.3-tikaparser1.patch, 
> NUTCH-961-1.4-dombuilder-1.patch, NUTCH-961-1.5-1.patch, 
> NUTCH-961-1.8-1.patch, NUTCH-961-2.1-v1.patch, NUTCH-961-2.1-v2.patch, 
> NUTCH-961v2.patch, nutch-2.x-boilerpipe.patch
>
>
> Tika 0.8 comes with the Boilerpipe content handler which can be used to 
> extract boilerplate content from HTML pages. We should see how we can expose 
> Boilerplate in the Nutch cofiguration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2016-01-24 Thread Tien Nguyen Manh (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15114658#comment-15114658
 ] 

Tien Nguyen Manh commented on NUTCH-961:


One note with boilerpipe support, it is significant slower than parse-html. I 
tested to parse the same segment and here are results
parse-html: 3hm, parse-tika with boilerpipe 5h10m and parse-tika without 
poilerpipe 4h.

> Expose Tika's boilerpipe support
> 
>
> Key: NUTCH-961
> URL: https://issues.apache.org/jira/browse/NUTCH-961
> Project: Nutch
>  Issue Type: New Feature
>  Components: parser
>Reporter: Markus Jelsma
>Assignee: Markus Jelsma
> Attachments: BoilerpipeExtractorRepository.java, 
> NUTCH-961-1.11-1.patch, NUTCH-961-1.3-3.patch, 
> NUTCH-961-1.3-tikaparser.patch, NUTCH-961-1.3-tikaparser1.patch, 
> NUTCH-961-1.4-dombuilder-1.patch, NUTCH-961-1.5-1.patch, 
> NUTCH-961-1.8-1.patch, NUTCH-961-2.1-v1.patch, NUTCH-961-2.1-v2.patch, 
> NUTCH-961v2.patch, nutch-2.x-boilerpipe.patch
>
>
> Tika 0.8 comes with the Boilerpipe content handler which can be used to 
> extract boilerplate content from HTML pages. We should see how we can expose 
> Boilerplate in the Nutch cofiguration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2016-01-21 Thread Markus Jelsma (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110373#comment-15110373
 ] 

Markus Jelsma commented on NUTCH-961:
-

Hello - that doesn't seem related to this issue as it doesn't interfere with 
how its loaded. Also, we cannot reproduce that locally nor in Hadoop mode. But 
there was some issue on the mailing list a couple of days ago that also 
mentioned an issue as you describe. 

> Expose Tika's boilerpipe support
> 
>
> Key: NUTCH-961
> URL: https://issues.apache.org/jira/browse/NUTCH-961
> Project: Nutch
>  Issue Type: New Feature
>  Components: parser
>Reporter: Markus Jelsma
>Assignee: Markus Jelsma
> Attachments: BoilerpipeExtractorRepository.java, 
> NUTCH-961-1.11-1.patch, NUTCH-961-1.3-3.patch, 
> NUTCH-961-1.3-tikaparser.patch, NUTCH-961-1.3-tikaparser1.patch, 
> NUTCH-961-1.4-dombuilder-1.patch, NUTCH-961-1.5-1.patch, 
> NUTCH-961-1.8-1.patch, NUTCH-961-2.1-v1.patch, NUTCH-961-2.1-v2.patch, 
> NUTCH-961v2.patch, nutch-2.x-boilerpipe.patch
>
>
> Tika 0.8 comes with the Boilerpipe content handler which can be used to 
> extract boilerplate content from HTML pages. We should see how we can expose 
> Boilerplate in the Nutch cofiguration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2016-01-21 Thread Markus Jelsma (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111292#comment-15111292
 ] 

Markus Jelsma commented on NUTCH-961:
-

Some news, the upstream Tika issue has been committed and resolved and i have 
requested an earlier Tika RC at which Chris Mattmann responded positive. An 
early Tika 1.12 might come soon after which i can quickly resolve NUTCH-1233 
and, of course, this issue.

One question to all of you and the PMC specifically, i would like to propose to 
enable Boilerpipe ArticleExtractor by default. I cannot think of any scenario 
at which a user would not want this. Please share your thoughts. :)

> Expose Tika's boilerpipe support
> 
>
> Key: NUTCH-961
> URL: https://issues.apache.org/jira/browse/NUTCH-961
> Project: Nutch
>  Issue Type: New Feature
>  Components: parser
>Reporter: Markus Jelsma
>Assignee: Markus Jelsma
> Attachments: BoilerpipeExtractorRepository.java, 
> NUTCH-961-1.11-1.patch, NUTCH-961-1.3-3.patch, 
> NUTCH-961-1.3-tikaparser.patch, NUTCH-961-1.3-tikaparser1.patch, 
> NUTCH-961-1.4-dombuilder-1.patch, NUTCH-961-1.5-1.patch, 
> NUTCH-961-1.8-1.patch, NUTCH-961-2.1-v1.patch, NUTCH-961-2.1-v2.patch, 
> NUTCH-961v2.patch, nutch-2.x-boilerpipe.patch
>
>
> Tika 0.8 comes with the Boilerpipe content handler which can be used to 
> extract boilerplate content from HTML pages. We should see how we can expose 
> Boilerplate in the Nutch cofiguration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2016-01-20 Thread Tien Nguyen Manh (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110217#comment-15110217
 ] 

Tien Nguyen Manh commented on NUTCH-961:


i'm using this patch NUTCH-961-1.11-1.patch, it works fine when run from 
eclipse & run in hadoop. It have problem when i run in local mode
It throws exception: "Can't retrieve Tika parser for mime-type text/html". It 
is not problem with parse-plugins.xml. It seem problem with TikaConfig 
constructor TikaConfig(ClassLoader loader), it failed to load some config via 
classLoader when run in local mode.

> Expose Tika's boilerpipe support
> 
>
> Key: NUTCH-961
> URL: https://issues.apache.org/jira/browse/NUTCH-961
> Project: Nutch
>  Issue Type: New Feature
>  Components: parser
>Reporter: Markus Jelsma
>Assignee: Markus Jelsma
> Attachments: BoilerpipeExtractorRepository.java, 
> NUTCH-961-1.11-1.patch, NUTCH-961-1.3-3.patch, 
> NUTCH-961-1.3-tikaparser.patch, NUTCH-961-1.3-tikaparser1.patch, 
> NUTCH-961-1.4-dombuilder-1.patch, NUTCH-961-1.5-1.patch, 
> NUTCH-961-1.8-1.patch, NUTCH-961-2.1-v1.patch, NUTCH-961-2.1-v2.patch, 
> NUTCH-961v2.patch, nutch-2.x-boilerpipe.patch
>
>
> Tika 0.8 comes with the Boilerpipe content handler which can be used to 
> extract boilerplate content from HTML pages. We should see how we can expose 
> Boilerplate in the Nutch cofiguration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2016-01-19 Thread Markus Jelsma (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106570#comment-15106570
 ] 

Markus Jelsma commented on NUTCH-961:
-

Yes but it requires NUTCH-1233. 

> Expose Tika's boilerpipe support
> 
>
> Key: NUTCH-961
> URL: https://issues.apache.org/jira/browse/NUTCH-961
> Project: Nutch
>  Issue Type: New Feature
>  Components: parser
>Reporter: Markus Jelsma
>Assignee: Markus Jelsma
> Attachments: BoilerpipeExtractorRepository.java, 
> NUTCH-961-1.11-1.patch, NUTCH-961-1.3-3.patch, 
> NUTCH-961-1.3-tikaparser.patch, NUTCH-961-1.3-tikaparser1.patch, 
> NUTCH-961-1.4-dombuilder-1.patch, NUTCH-961-1.5-1.patch, 
> NUTCH-961-1.8-1.patch, NUTCH-961-2.1-v1.patch, NUTCH-961-2.1-v2.patch, 
> NUTCH-961v2.patch, nutch-2.x-boilerpipe.patch
>
>
> Tika 0.8 comes with the Boilerpipe content handler which can be used to 
> extract boilerplate content from HTML pages. We should see how we can expose 
> Boilerplate in the Nutch cofiguration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2016-01-19 Thread Markus Jelsma (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106783#comment-15106783
 ] 

Markus Jelsma commented on NUTCH-961:
-

Update, i've updated NUTCH-1233 for current trunk as well as a fix for the 
outlink extraction in Tika via TIKA-1835.

> Expose Tika's boilerpipe support
> 
>
> Key: NUTCH-961
> URL: https://issues.apache.org/jira/browse/NUTCH-961
> Project: Nutch
>  Issue Type: New Feature
>  Components: parser
>Reporter: Markus Jelsma
>Assignee: Markus Jelsma
> Attachments: BoilerpipeExtractorRepository.java, 
> NUTCH-961-1.11-1.patch, NUTCH-961-1.3-3.patch, 
> NUTCH-961-1.3-tikaparser.patch, NUTCH-961-1.3-tikaparser1.patch, 
> NUTCH-961-1.4-dombuilder-1.patch, NUTCH-961-1.5-1.patch, 
> NUTCH-961-1.8-1.patch, NUTCH-961-2.1-v1.patch, NUTCH-961-2.1-v2.patch, 
> NUTCH-961v2.patch, nutch-2.x-boilerpipe.patch
>
>
> Tika 0.8 comes with the Boilerpipe content handler which can be used to 
> extract boilerplate content from HTML pages. We should see how we can expose 
> Boilerplate in the Nutch cofiguration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2016-01-18 Thread Otis Gospodnetic (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106152#comment-15106152
 ] 

Otis Gospodnetic commented on NUTCH-961:


Any chance we could commit this, [~markus.jel...@openindex.io]?

> Expose Tika's boilerpipe support
> 
>
> Key: NUTCH-961
> URL: https://issues.apache.org/jira/browse/NUTCH-961
> Project: Nutch
>  Issue Type: New Feature
>  Components: parser
>Reporter: Markus Jelsma
>Assignee: Markus Jelsma
> Attachments: BoilerpipeExtractorRepository.java, 
> NUTCH-961-1.11-1.patch, NUTCH-961-1.3-3.patch, 
> NUTCH-961-1.3-tikaparser.patch, NUTCH-961-1.3-tikaparser1.patch, 
> NUTCH-961-1.4-dombuilder-1.patch, NUTCH-961-1.5-1.patch, 
> NUTCH-961-1.8-1.patch, NUTCH-961-2.1-v1.patch, NUTCH-961-2.1-v2.patch, 
> NUTCH-961v2.patch, nutch-2.x-boilerpipe.patch
>
>
> Tika 0.8 comes with the Boilerpipe content handler which can be used to 
> extract boilerplate content from HTML pages. We should see how we can expose 
> Boilerplate in the Nutch cofiguration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2015-04-01 Thread Alexander Kingson (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391558#comment-14391558
 ] 

Alexander Kingson commented on NUTCH-961:
-

Hello,

Since I was not getting satisfactory results after upgrading to boilerpipe 
1.2.0 with parse-tika (with boilerpipe support)  I have put some code to 
nutch-2.x parser to get the same results as the boilerpipe demo-website. Used 
some code from .v2.patch. 
Attaching the patch.

Thanks.
Alex.

 Expose Tika's boilerpipe support
 

 Key: NUTCH-961
 URL: https://issues.apache.org/jira/browse/NUTCH-961
 Project: Nutch
  Issue Type: New Feature
  Components: parser
Reporter: Markus Jelsma
Assignee: Markus Jelsma
 Fix For: 1.11

 Attachments: BoilerpipeExtractorRepository.java, 
 NUTCH-961-1.3-3.patch, NUTCH-961-1.3-tikaparser.patch, 
 NUTCH-961-1.3-tikaparser1.patch, NUTCH-961-1.4-dombuilder-1.patch, 
 NUTCH-961-1.5-1.patch, NUTCH-961-1.8-1.patch, NUTCH-961-2.1-v1.patch, 
 NUTCH-961-2.1-v2.patch, NUTCH-961v2.patch


 Tika 0.8 comes with the Boilerpipe content handler which can be used to 
 extract boilerplate content from HTML pages. We should see how we can expose 
 Boilerplate in the Nutch cofiguration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2014-02-13 Thread Markus Jelsma (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13900180#comment-13900180
 ] 

Markus Jelsma commented on NUTCH-961:
-

I am sorry, i did not mean to speak for the Nutch PMC at all; we not using BP 
means I am not using BP. As i said before, i am happy to commit this issue is 
the linked issues are resolved first.

 Expose Tika's boilerpipe support
 

 Key: NUTCH-961
 URL: https://issues.apache.org/jira/browse/NUTCH-961
 Project: Nutch
  Issue Type: New Feature
  Components: parser
Reporter: Markus Jelsma
Assignee: Markus Jelsma
 Fix For: 2.3, 1.8

 Attachments: BoilerpipeExtractorRepository.java, 
 NUTCH-961-1.3-3.patch, NUTCH-961-1.3-tikaparser.patch, 
 NUTCH-961-1.3-tikaparser1.patch, NUTCH-961-1.4-dombuilder-1.patch, 
 NUTCH-961-1.5-1.patch, NUTCH-961-1.8-1.patch, NUTCH-961-2.1-v1.patch, 
 NUTCH-961-2.1-v2.patch, NUTCH-961v2.patch


 Tika 0.8 comes with the Boilerpipe content handler which can be used to 
 extract boilerplate content from HTML pages. We should see how we can expose 
 Boilerplate in the Nutch cofiguration.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2014-02-12 Thread Matzz (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899044#comment-13899044
 ] 

Matzz commented on NUTCH-961:
-

{quote}We don't use it BP anymore {quote}

BP integration will be totally abandoned? Are there any plans to use other 
content extractor in favour of Boilerpipe?

 Expose Tika's boilerpipe support
 

 Key: NUTCH-961
 URL: https://issues.apache.org/jira/browse/NUTCH-961
 Project: Nutch
  Issue Type: New Feature
  Components: parser
Reporter: Markus Jelsma
Assignee: Markus Jelsma
 Fix For: 2.3, 1.8

 Attachments: BoilerpipeExtractorRepository.java, 
 NUTCH-961-1.3-3.patch, NUTCH-961-1.3-tikaparser.patch, 
 NUTCH-961-1.3-tikaparser1.patch, NUTCH-961-1.4-dombuilder-1.patch, 
 NUTCH-961-1.5-1.patch, NUTCH-961-1.8-1.patch, NUTCH-961-2.1-v1.patch, 
 NUTCH-961-2.1-v2.patch, NUTCH-961v2.patch


 Tika 0.8 comes with the Boilerpipe content handler which can be used to 
 extract boilerplate content from HTML pages. We should see how we can expose 
 Boilerplate in the Nutch cofiguration.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2013-10-08 Thread Otis Gospodnetic (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13789686#comment-13789686
 ] 

Otis Gospodnetic commented on NUTCH-961:


Looks like [~kkrugler] is offering to help with publishing Boilerpipe to a 
Sonatype Maven repo in TIKA-676 (this Nutch issue apparently depends on this 
Tika issue) - thanks Ken!

But note that simply moving Nutch to Boilerpipe 1.2.0 won't fix the issue 
[~tiennm] just reported.
[~markus17], if [~tiennm] provides a patch that makes Nutch Boilerpipe output 
match that of the Boilerpipe demo, could you commit it to 2.x?

 Expose Tika's boilerpipe support
 

 Key: NUTCH-961
 URL: https://issues.apache.org/jira/browse/NUTCH-961
 Project: Nutch
  Issue Type: New Feature
  Components: parser
Reporter: Markus Jelsma
Assignee: Markus Jelsma
 Fix For: 2.3, 1.8

 Attachments: BoilerpipeExtractorRepository.java, 
 NUTCH-961-1.3-3.patch, NUTCH-961-1.3-tikaparser1.patch, 
 NUTCH-961-1.3-tikaparser.patch, NUTCH-961-1.4-dombuilder-1.patch, 
 NUTCH-961-1.5-1.patch, NUTCH-961-1.8-1.patch, NUTCH-961-2.1-v1.patch, 
 NUTCH-961-2.1-v2.patch, NUTCH-961v2.patch


 Tika 0.8 comes with the Boilerpipe content handler which can be used to 
 extract boilerplate content from HTML pages. We should see how we can expose 
 Boilerplate in the Nutch cofiguration.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2013-10-08 Thread Markus Jelsma (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13789735#comment-13789735
 ] 

Markus Jelsma commented on NUTCH-961:
-

Hi Otis - there are no significant improvements between the 1.1.0 and 1.2.0 of 
Boilerpipe, at least not when it comes to better extraction. I am very sure 
that when the demo was using 1.2.0, we got identical results with 1.2.0 as 
well, but still poor in cases not suitable such as overviews, blocks etc. I am 
also very sure that the current 1.2.0 is nowadays different than what the demo 
returns, it is not identical anymore, and improved quite a lot.

We don't use it BP anymore but i'm happy to commit whenever 1.2.0 is in maven 
or part of Tika if it gets donated to the ASF. We need to get NUTCH-1233 in as 
well then.

 Expose Tika's boilerpipe support
 

 Key: NUTCH-961
 URL: https://issues.apache.org/jira/browse/NUTCH-961
 Project: Nutch
  Issue Type: New Feature
  Components: parser
Reporter: Markus Jelsma
Assignee: Markus Jelsma
 Fix For: 2.3, 1.8

 Attachments: BoilerpipeExtractorRepository.java, 
 NUTCH-961-1.3-3.patch, NUTCH-961-1.3-tikaparser1.patch, 
 NUTCH-961-1.3-tikaparser.patch, NUTCH-961-1.4-dombuilder-1.patch, 
 NUTCH-961-1.5-1.patch, NUTCH-961-1.8-1.patch, NUTCH-961-2.1-v1.patch, 
 NUTCH-961-2.1-v2.patch, NUTCH-961v2.patch


 Tika 0.8 comes with the Boilerpipe content handler which can be used to 
 extract boilerplate content from HTML pages. We should see how we can expose 
 Boilerplate in the Nutch cofiguration.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2013-10-08 Thread Otis Gospodnetic (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13789894#comment-13789894
 ] 

Otis Gospodnetic commented on NUTCH-961:


bq. We don't use it BP anymore

What do you mean by that?  I looked at parse-tika/plugins.xml earlier today and 
saw BP 1.1.0 there.  So I'm not sure what you mean...

 Expose Tika's boilerpipe support
 

 Key: NUTCH-961
 URL: https://issues.apache.org/jira/browse/NUTCH-961
 Project: Nutch
  Issue Type: New Feature
  Components: parser
Reporter: Markus Jelsma
Assignee: Markus Jelsma
 Fix For: 2.3, 1.8

 Attachments: BoilerpipeExtractorRepository.java, 
 NUTCH-961-1.3-3.patch, NUTCH-961-1.3-tikaparser1.patch, 
 NUTCH-961-1.3-tikaparser.patch, NUTCH-961-1.4-dombuilder-1.patch, 
 NUTCH-961-1.5-1.patch, NUTCH-961-1.8-1.patch, NUTCH-961-2.1-v1.patch, 
 NUTCH-961-2.1-v2.patch, NUTCH-961v2.patch


 Tika 0.8 comes with the Boilerpipe content handler which can be used to 
 extract boilerplate content from HTML pages. We should see how we can expose 
 Boilerplate in the Nutch cofiguration.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2013-10-07 Thread Nguyen Manh Tien (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13788911#comment-13788911
 ] 

Nguyen Manh Tien commented on NUTCH-961:


I used patch NUTCH-961-2.1-v2.patch for nutch-2.2.1
i found that the text parsed by nutch-tika (with boilerpipe support) is 
different from text parsed by demo site http://boilerpipe-web.appspot.com 
I did upgrade to boilerpipe 1.2.0 to be match with demo site.

The url i tested is http://www.medhelp.org/posts/Eye-Care/EYE/show/1199003

The text from nutch-tika (i use ArticleExtractor)

EYE - Eye Care - MedHelp Experts My MedHelp Login or Signup Eye Care Community 
EYE Post a Question « Back to Community About This Community: This patient 
support community is for discussions relating to eye care, cataracts , glaucoma 
, retinal detachment , eye infections, misaligned eyes , intra-ocular implants, 
refractive surgery ( LASIK and CK), glasses, contact lenses, amblyopia , eye 
injuries, dry eyes , ocular allergy, eye pain and discomfort, pediatric eye 
disorders, eyelid and tearduct surgery, poor eyesight, and eye surgery. View 
community archives Font Size: A A ABackground: Search this Community: Go 3 
Comments EYE My son is 4 and half years old and have + no .Our doctor told me 
six months ago that + no. decreases as time passed and he not to wear glasses 
after two -three years if he wears glasses regularly.But yesterday he told me 
that his + No. increases and he have to wear glasses always.If you wish u can 
go for laser surgery after 14 years i.e. when my son will have age of 17 
years.please help me what to do ? Watch this discussion Tweet Related 
Discussions How to decide if glasses are needed for children? (8 replies):How 
can a Doctor tell if a child has amblyopia? Is t... [more] Astigmatism (1 
replies):My 5 year old son has severe astigmatism. He wears glass... [more] Can 
someone help me in regards to my sons eyes? (6 replies):I had noticed my son 
had, had an eye issue when he was a... [more] Blurred vision with glasses (2 
replies):Hi, I recently got new glasses and but the vision in my ... [more] 
Eyesight getting worse (2 replies):Hello! So here's the story. My eyesight had 
never been ... [more]

AND from demo

3 Comments
EYE
My son is 4 and half years old and have + no .Our doctor told me six months ago 
that + no. decreases as time passed and he not to wear glasses after two -three 
years if he wears glasses regularly.But yesterday he told me that his + No. 
increases and he have to wear glasses always.If you wish u can go for laser 
surgery after 14 years i.e. when my son will have age of 17 years.please help 
me what to do ?

the result from demo is much better for this url.
So the parse-tike/boilerpipe not only extract main content from page but also 
include title and other node content.
Is it expected?

 Expose Tika's boilerpipe support
 

 Key: NUTCH-961
 URL: https://issues.apache.org/jira/browse/NUTCH-961
 Project: Nutch
  Issue Type: New Feature
  Components: parser
Reporter: Markus Jelsma
Assignee: Markus Jelsma
 Fix For: 2.3, 1.8

 Attachments: BoilerpipeExtractorRepository.java, 
 NUTCH-961-1.3-3.patch, NUTCH-961-1.3-tikaparser1.patch, 
 NUTCH-961-1.3-tikaparser.patch, NUTCH-961-1.4-dombuilder-1.patch, 
 NUTCH-961-1.5-1.patch, NUTCH-961-1.8-1.patch, NUTCH-961-2.1-v1.patch, 
 NUTCH-961-2.1-v2.patch, NUTCH-961v2.patch


 Tika 0.8 comes with the Boilerpipe content handler which can be used to 
 extract boilerplate content from HTML pages. We should see how we can expose 
 Boilerplate in the Nutch cofiguration.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2013-03-29 Thread Miles Rowland (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13617739#comment-13617739
 ] 

Miles Rowland commented on NUTCH-961:
-

Roland, thanks for porting to 2.1. I'm having an issue where nutch is only 
successfully parsing the first fetched url, and all other urls fail to parse 
with a warning unable to successfully parse content [website] of type [x]. If 
I run parseChecker on that url the parse runs successfully using 
tika/boilerplate, so it seems to be an issue that only occurs when trying to 
run the second parse or more in a batch job. 

I'm running Nutch 2.1 with MySQL. The problem occurs with both bp1.1.0 and 
1.2.0. 

 Expose Tika's boilerpipe support
 

 Key: NUTCH-961
 URL: https://issues.apache.org/jira/browse/NUTCH-961
 Project: Nutch
  Issue Type: New Feature
  Components: parser
Reporter: Markus Jelsma
Assignee: Markus Jelsma
 Fix For: 1.7, 2.2

 Attachments: BoilerpipeExtractorRepository.java, 
 NUTCH-961-1.3-3.patch, NUTCH-961-1.3-tikaparser1.patch, 
 NUTCH-961-1.3-tikaparser.patch, NUTCH-961-1.4-dombuilder-1.patch, 
 NUTCH-961-1.5-1.patch, NUTCH-961-2.1-v1.patch, NUTCH-961-2.1-v2.patch, 
 NUTCH-961v2.patch


 Tika 0.8 comes with the Boilerpipe content handler which can be used to 
 extract boilerplate content from HTML pages. We should see how we can expose 
 Boilerplate in the Nutch cofiguration.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2013-03-04 Thread Roland (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13592056#comment-13592056
 ] 

Roland commented on NUTCH-961:
--

Kiran, did you already start porting it to 2.x?

 Expose Tika's boilerpipe support
 

 Key: NUTCH-961
 URL: https://issues.apache.org/jira/browse/NUTCH-961
 Project: Nutch
  Issue Type: New Feature
  Components: parser
Reporter: Markus Jelsma
Assignee: Markus Jelsma
 Fix For: 1.7

 Attachments: BoilerpipeExtractorRepository.java, 
 NUTCH-961-1.3-3.patch, NUTCH-961-1.3-tikaparser1.patch, 
 NUTCH-961-1.3-tikaparser.patch, NUTCH-961-1.4-dombuilder-1.patch, 
 NUTCH-961-1.5-1.patch, NUTCH-961v2.patch


 Tika 0.8 comes with the Boilerpipe content handler which can be used to 
 extract boilerplate content from HTML pages. We should see how we can expose 
 Boilerplate in the Nutch cofiguration.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2013-03-04 Thread kiran (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13592179#comment-13592179
 ] 

kiran commented on NUTCH-961:
-

No Roland, not yet. I just switched to using 1.x series, but i will give a try 
at porting this to 2.x this week

 Expose Tika's boilerpipe support
 

 Key: NUTCH-961
 URL: https://issues.apache.org/jira/browse/NUTCH-961
 Project: Nutch
  Issue Type: New Feature
  Components: parser
Reporter: Markus Jelsma
Assignee: Markus Jelsma
 Fix For: 1.7

 Attachments: BoilerpipeExtractorRepository.java, 
 NUTCH-961-1.3-3.patch, NUTCH-961-1.3-tikaparser1.patch, 
 NUTCH-961-1.3-tikaparser.patch, NUTCH-961-1.4-dombuilder-1.patch, 
 NUTCH-961-1.5-1.patch, NUTCH-961v2.patch


 Tika 0.8 comes with the Boilerpipe content handler which can be used to 
 extract boilerplate content from HTML pages. We should see how we can expose 
 Boilerplate in the Nutch cofiguration.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: [jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2013-03-04 Thread Roland 

Hey Kiran,

drop me a line prior to starting, I will give it a try tomorrow (I hope).

--Roland

Am 04.03.2013 14:13, schrieb kiran (JIRA):

 [ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13592179#comment-13592179
 ]

kiran commented on NUTCH-961:
-

No Roland, not yet. I just switched to using 1.x series, but i will give a try 
at porting this to 2.x this week
 

Expose Tika's boilerpipe support


 Key: NUTCH-961
 URL: https://issues.apache.org/jira/browse/NUTCH-961
 Project: Nutch
  Issue Type: New Feature
  Components: parser
Reporter: Markus Jelsma
Assignee: Markus Jelsma
 Fix For: 1.7

 Attachments: BoilerpipeExtractorRepository.java, 
NUTCH-961-1.3-3.patch, NUTCH-961-1.3-tikaparser1.patch, 
NUTCH-961-1.3-tikaparser.patch, NUTCH-961-1.4-dombuilder-1.patch, 
NUTCH-961-1.5-1.patch, NUTCH-961v2.patch


Tika 0.8 comes with the Boilerpipe content handler which can be used to extract 
boilerplate content from HTML pages. We should see how we can expose 
Boilerplate in the Nutch cofiguration.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2013-02-19 Thread kiran (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13581459#comment-13581459
 ] 

kiran commented on NUTCH-961:
-

Markus, do you think this patch can also work for 2.x Series ? If not, is it 
easy to port to 2.x ? Please let me know your suggestions.

 Expose Tika's boilerpipe support
 

 Key: NUTCH-961
 URL: https://issues.apache.org/jira/browse/NUTCH-961
 Project: Nutch
  Issue Type: New Feature
  Components: parser
Reporter: Markus Jelsma
Assignee: Markus Jelsma
 Fix For: 1.7

 Attachments: BoilerpipeExtractorRepository.java, 
 NUTCH-961-1.3-3.patch, NUTCH-961-1.3-tikaparser1.patch, 
 NUTCH-961-1.3-tikaparser.patch, NUTCH-961-1.4-dombuilder-1.patch, 
 NUTCH-961-1.5-1.patch, NUTCH-961v2.patch


 Tika 0.8 comes with the Boilerpipe content handler which can be used to 
 extract boilerplate content from HTML pages. We should see how we can expose 
 Boilerplate in the Nutch cofiguration.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2013-02-19 Thread Markus Jelsma (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13581530#comment-13581530
 ] 

Markus Jelsma commented on NUTCH-961:
-

Should work fine, parse plugins have not changed that much. Keep in mind that 
you may need bp1.2.0 and keep an eye on link extraction. See related issues.

 Expose Tika's boilerpipe support
 

 Key: NUTCH-961
 URL: https://issues.apache.org/jira/browse/NUTCH-961
 Project: Nutch
  Issue Type: New Feature
  Components: parser
Reporter: Markus Jelsma
Assignee: Markus Jelsma
 Fix For: 1.7

 Attachments: BoilerpipeExtractorRepository.java, 
 NUTCH-961-1.3-3.patch, NUTCH-961-1.3-tikaparser1.patch, 
 NUTCH-961-1.3-tikaparser.patch, NUTCH-961-1.4-dombuilder-1.patch, 
 NUTCH-961-1.5-1.patch, NUTCH-961v2.patch


 Tika 0.8 comes with the Boilerpipe content handler which can be used to 
 extract boilerplate content from HTML pages. We should see how we can expose 
 Boilerplate in the Nutch cofiguration.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2011-12-27 Thread Markus Jelsma (Commented) (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13176194#comment-13176194
 ] 

Markus Jelsma commented on NUTCH-961:
-

Fixed already. See NUTCH-1233 for a patch!

 Expose Tika's boilerpipe support
 

 Key: NUTCH-961
 URL: https://issues.apache.org/jira/browse/NUTCH-961
 Project: Nutch
  Issue Type: New Feature
  Components: parser
Reporter: Markus Jelsma
Assignee: Markus Jelsma
 Fix For: 1.5

 Attachments: BoilerpipeExtractorRepository.java, 
 NUTCH-961-1.3-3.patch, NUTCH-961-1.3-tikaparser.patch, 
 NUTCH-961-1.3-tikaparser1.patch, NUTCH-961-1.4-dombuilder-1.patch, 
 NUTCH-961-1.5-1.patch, NUTCH-961v2.patch


 Tika 0.8 comes with the Boilerpipe content handler which can be used to 
 extract boilerplate content from HTML pages. We should see how we can expose 
 Boilerplate in the Nutch cofiguration.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2011-06-10 Thread Gabriele Kahlout (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13047130#comment-13047130
 ] 

Gabriele Kahlout commented on NUTCH-961:


{quote}it needs to use a different ContentHandler in parse-tika itself.{quote}
[Documentation opportunity] why?

My intuition is that the default sax ContentHandler returns the full page and 
then Tika handles it, this time with the boilerpipe option. 

 Expose Tika's boilerpipe support
 

 Key: NUTCH-961
 URL: https://issues.apache.org/jira/browse/NUTCH-961
 Project: Nutch
  Issue Type: New Feature
  Components: parser
Reporter: Markus Jelsma
 Fix For: 1.4, 2.0

 Attachments: BoilerpipeExtractorRepository.java, 
 NUTCH-961-1.3-tikaparser.patch, NUTCH-961-1.3-tikaparser1.patch, 
 NUTCH-961v2.patch


 Tika 0.8 comes with the Boilerpipe content handler which can be used to 
 extract boilerplate content from HTML pages. We should see how we can expose 
 Boilerplate in the Nutch cofiguration.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2011-06-10 Thread Ken Krugler (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13047490#comment-13047490
 ] 

Ken Krugler commented on NUTCH-961:
---

The way that Boilerpipe in Tika works is that it acts as a delegate, processing 
the SAX events generated by the default content handler that knows how to help 
clean up broken HTML.

So it's incremental processing (you don't need to get the full page first).

Separate note: Tika's Boilerpipe support now has an option to return HTML 
markup, so you could run it in this mode to get anchors/anchor text.


 Expose Tika's boilerpipe support
 

 Key: NUTCH-961
 URL: https://issues.apache.org/jira/browse/NUTCH-961
 Project: Nutch
  Issue Type: New Feature
  Components: parser
Reporter: Markus Jelsma
 Fix For: 1.4, 2.0

 Attachments: BoilerpipeExtractorRepository.java, 
 NUTCH-961-1.3-tikaparser.patch, NUTCH-961-1.3-tikaparser1.patch, 
 NUTCH-961v2.patch


 Tika 0.8 comes with the Boilerpipe content handler which can be used to 
 extract boilerplate content from HTML pages. We should see how we can expose 
 Boilerplate in the Nutch cofiguration.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2011-06-10 Thread Markus Jelsma (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13047501#comment-13047501
 ] 

Markus Jelsma commented on NUTCH-961:
-

Ah, that's great! Is this in 0.9 or trunk? We still bind with 0.9. This may be  
useful because this patch doesn't add anchors to the detected outlinks. The 
last anchor(s) may contain the complete BP body! =D

 Expose Tika's boilerpipe support
 

 Key: NUTCH-961
 URL: https://issues.apache.org/jira/browse/NUTCH-961
 Project: Nutch
  Issue Type: New Feature
  Components: parser
Reporter: Markus Jelsma
 Fix For: 1.4, 2.0

 Attachments: BoilerpipeExtractorRepository.java, 
 NUTCH-961-1.3-tikaparser.patch, NUTCH-961-1.3-tikaparser1.patch, 
 NUTCH-961v2.patch


 Tika 0.8 comes with the Boilerpipe content handler which can be used to 
 extract boilerplate content from HTML pages. We should see how we can expose 
 Boilerplate in the Nutch cofiguration.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2011-04-26 Thread Gabriele Kahlout (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13025286#comment-13025286
 ] 

Gabriele Kahlout commented on NUTCH-961:


@Markus - Thank you.

Watch out for [1] in parse-plugins.xml. .html pages may indeed by xhtml. You 
can safely delete alla parse-html mimeType associations, as long as you have 
[2] (and you want to use parse-tika instead of parse-html ).

[1]
mimeType name=application/xhtml+xml
plugin id=parse-html /
/mimeType

[2] 
!--  by default if the mimeType is set to *, or 
if it can't be determined, use parse-tika --
mimeType name=*
  plugin id=parse-tika /
/mimeType
 

 Expose Tika's boilerpipe support
 

 Key: NUTCH-961
 URL: https://issues.apache.org/jira/browse/NUTCH-961
 Project: Nutch
  Issue Type: New Feature
  Components: parser
Reporter: Markus Jelsma
 Fix For: 2.0

 Attachments: BoilerpipeExtractorRepository.java, 
 NUTCH-961-1.3-tikaparser.patch


 Tika 0.8 comes with the Boilerpipe content handler which can be used to 
 extract boilerplate content from HTML pages. We should see how we can expose 
 Boilerplate in the Nutch cofiguration.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (NUTCH-961) Expose Tika's boilerpipe support

2011-04-26 Thread Markus Jelsma (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13025295#comment-13025295
 ] 

Markus Jelsma commented on NUTCH-961:
-

Not safely, there are still issues regarding HTML parsing with Tika, even 
without this nasty boilerpipe hack.

 Expose Tika's boilerpipe support
 

 Key: NUTCH-961
 URL: https://issues.apache.org/jira/browse/NUTCH-961
 Project: Nutch
  Issue Type: New Feature
  Components: parser
Reporter: Markus Jelsma
 Fix For: 2.0

 Attachments: BoilerpipeExtractorRepository.java, 
 NUTCH-961-1.3-tikaparser.patch


 Tika 0.8 comes with the Boilerpipe content handler which can be used to 
 extract boilerplate content from HTML pages. We should see how we can expose 
 Boilerplate in the Nutch cofiguration.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (NUTCH-961) Expose Tika's boilerpipe support

2011-01-27 Thread Markus Jelsma (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12987575#action_12987575
 ] 

Markus Jelsma commented on NUTCH-961:
-

Boilerpipe comes with several algorithms for stripping away the boilerplate 
content. Although the ArticleExtractor is recommended, it certainly fails for 
many types of pages. Pages such as news overviews with blocks and lists are 
much better extracted with the CanolaExtractor instead. This poses a problem, 
we cannot have just one single configuration directive telling the parser which 
extractor to use for a whole crawl.

Some thoughts on how to deal with it:
- use Boilerpipe's estimator to automatically determine which extractor to use
- have a facility to override false positives returned by the estimator and 
hardcode which extractor to use for URL groups (not unlike the subcollection 
plugin)


 Expose Tika's boilerpipe support
 

 Key: NUTCH-961
 URL: https://issues.apache.org/jira/browse/NUTCH-961
 Project: Nutch
  Issue Type: New Feature
  Components: parser
Reporter: Markus Jelsma
 Fix For: 2.0


 Tika 0.8 comes with the Boilerpipe content handler which can be used to 
 extract boilerplate content from HTML pages. We should see how we can expose 
 Boilerplate in the Nutch cofiguration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.