[jira] [Commented] (NUTCH-2102) WARC Exporter
[ https://issues.apache.org/jira/browse/NUTCH-2102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14902752#comment-14902752 ] Hudson commented on NUTCH-2102: --- SUCCESS: Integrated in Nutch-trunk #3278 (See [https://builds.apache.org/job/Nutch-trunk/3278/]) NUTCH-2102 WARC Exporter (jnioche: http://svn.apache.org/viewvc/nutch/trunk/?view=rev&rev=1704634) * /nutch/trunk/CHANGES.txt * /nutch/trunk/ivy/ivy.xml * /nutch/trunk/src/bin/nutch * /nutch/trunk/src/java/org/apache/nutch/tools/warc * /nutch/trunk/src/java/org/apache/nutch/tools/warc/WARCExporter.java * /nutch/trunk/src/java/org/apache/nutch/tools/warc/package-info.java > WARC Exporter > - > > Key: NUTCH-2102 > URL: https://issues.apache.org/jira/browse/NUTCH-2102 > Project: Nutch > Issue Type: Improvement > Components: commoncrawl, dumpers >Affects Versions: 1.10 >Reporter: Julien Nioche > Fix For: 1.11 > > Attachments: NUTCH-2102.patch > > > This patch adds a WARC exporter > [http://bibnum.bnf.fr/warc/WARC_ISO_28500_version1_latestdraft.pdf]. Unlike > the code submitted in [https://github.com/apache/nutch/pull/55] which is > based on the CommonCrawlDataDumper, this exporter is a MapReduce job and > hence should be able to cope with large segments in a timely fashion and also > is not limited to the local file system. > Later on we could have a WARCImporter to generate segments from WARC files, > which is outside the scope of the CCDD anyway. Also WARC is not specific to > CommonCrawl, which is why the package name does not reflect it. > I don't think it would be a problem to have both the modified CCDD and this > class providing similar functionalities. > This class is called in the following way > ./nutch org.apache.nutch.tools.warc.WARCExporter > /data/nutch-dipe/1kcrawl/warc -dir /data/nutch-dipe/1kcrawl/segments/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NUTCH-2102) WARC Exporter
[ https://issues.apache.org/jira/browse/NUTCH-2102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14875754#comment-14875754 ] Jorge Luis Betancourt Gonzalez commented on NUTCH-2102: --- +1 It looks good, the nutch entry will definitively will make it easier to use :) > WARC Exporter > - > > Key: NUTCH-2102 > URL: https://issues.apache.org/jira/browse/NUTCH-2102 > Project: Nutch > Issue Type: Improvement > Components: commoncrawl, dumpers >Affects Versions: 1.10 >Reporter: Julien Nioche > Attachments: NUTCH-2102.patch > > > This patch adds a WARC exporter > [http://bibnum.bnf.fr/warc/WARC_ISO_28500_version1_latestdraft.pdf]. Unlike > the code submitted in [https://github.com/apache/nutch/pull/55] which is > based on the CommonCrawlDataDumper, this exporter is a MapReduce job and > hence should be able to cope with large segments in a timely fashion and also > is not limited to the local file system. > Later on we could have a WARCImporter to generate segments from WARC files, > which is outside the scope of the CCDD anyway. Also WARC is not specific to > CommonCrawl, which is why the package name does not reflect it. > I don't think it would be a problem to have both the modified CCDD and this > class providing similar functionalities. > This class is called in the following way > ./nutch org.apache.nutch.tools.warc.WARCExporter > /data/nutch-dipe/1kcrawl/warc -dir /data/nutch-dipe/1kcrawl/segments/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NUTCH-2102) WARC Exporter
[ https://issues.apache.org/jira/browse/NUTCH-2102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14747327#comment-14747327 ] Julien Nioche commented on NUTCH-2102: -- Hi Markus > I believe this warc format is the updated arc format, for which we already > have an importer? The importer for WARC could be done later on and would leverage the same library as the exporter. But yes, it would look pretty similar to the ARC one . > you meant to use StringBuilder instead of the synchronized StringBuffer in > HttpResponse could do, will wait for other comments before amending the patch > A bin/nutch entry is also missing, or not yes, why not. There's already far too much stuff in there :-) though. Again, I can amend it if ppl are +1 for committing this > WARC Exporter > - > > Key: NUTCH-2102 > URL: https://issues.apache.org/jira/browse/NUTCH-2102 > Project: Nutch > Issue Type: Improvement > Components: commoncrawl, dumpers >Affects Versions: 1.10 >Reporter: Julien Nioche > Attachments: NUTCH-2102.patch > > > This patch adds a WARC exporter > [http://bibnum.bnf.fr/warc/WARC_ISO_28500_version1_latestdraft.pdf]. Unlike > the code submitted in [https://github.com/apache/nutch/pull/55] which is > based on the CommonCrawlDataDumper, this exporter is a MapReduce job and > hence should be able to cope with large segments in a timely fashion and also > is not limited to the local file system. > Later on we could have a WARCImporter to generate segments from WARC files, > which is outside the scope of the CCDD anyway. Also WARC is not specific to > CommonCrawl, which is why the package name does not reflect it. > I don't think it would be a problem to have both the modified CCDD and this > class providing similar functionalities. > This class is called in the following way > ./nutch org.apache.nutch.tools.warc.WARCExporter > /data/nutch-dipe/1kcrawl/warc -dir /data/nutch-dipe/1kcrawl/segments/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NUTCH-2102) WARC Exporter
[ https://issues.apache.org/jira/browse/NUTCH-2102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14747316#comment-14747316 ] Markus Jelsma commented on NUTCH-2102: -- Hello Julien! I believe this warc format is the updated arc format, for which we already have an importer? Anyway, code seems fine although i think you meant to use StringBuilder instead of the synchronized StringBuffer in HttpResponse. A bin/nutch entry is also missing, or not :) > WARC Exporter > - > > Key: NUTCH-2102 > URL: https://issues.apache.org/jira/browse/NUTCH-2102 > Project: Nutch > Issue Type: Improvement > Components: commoncrawl, dumpers >Affects Versions: 1.10 >Reporter: Julien Nioche > Attachments: NUTCH-2102.patch > > > This patch adds a WARC exporter > [http://bibnum.bnf.fr/warc/WARC_ISO_28500_version1_latestdraft.pdf]. Unlike > the code submitted in [https://github.com/apache/nutch/pull/55] which is > based on the CommonCrawlDataDumper, this exporter is a MapReduce job and > hence should be able to cope with large segments in a timely fashion and also > is not limited to the local file system. > Later on we could have a WARCImporter to generate segments from WARC files, > which is outside the scope of the CCDD anyway. Also WARC is not specific to > CommonCrawl, which is why the package name does not reflect it. > I don't think it would be a problem to have both the modified CCDD and this > class providing similar functionalities. > This class is called in the following way > ./nutch org.apache.nutch.tools.warc.WARCExporter > /data/nutch-dipe/1kcrawl/warc -dir /data/nutch-dipe/1kcrawl/segments/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NUTCH-2102) WARC Exporter
[ https://issues.apache.org/jira/browse/NUTCH-2102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14747301#comment-14747301 ] Julien Nioche commented on NUTCH-2102: -- Please review > WARC Exporter > - > > Key: NUTCH-2102 > URL: https://issues.apache.org/jira/browse/NUTCH-2102 > Project: Nutch > Issue Type: Improvement > Components: commoncrawl, dumpers >Affects Versions: 1.10 >Reporter: Julien Nioche > Attachments: NUTCH-2102.patch > > > This patch adds a WARC exporter > [http://bibnum.bnf.fr/warc/WARC_ISO_28500_version1_latestdraft.pdf]. Unlike > the code submitted in [https://github.com/apache/nutch/pull/55] which is > based on the CommonCrawlDataDumper, this exporter is a MapReduce job and > hence should be able to cope with large segments in a timely fashion and also > is not limited to the local file system. > Later on we could have a WARCImporter to generate segments from WARC files, > which is outside the scope of the CCDD anyway. Also WARC is not specific to > CommonCrawl, which is why the package name does not reflect it. > I don't think it would be a problem to have both the modified CCDD and this > class providing similar functionalities. > This class is called in the following way > ./nutch org.apache.nutch.tools.warc.WARCExporter > /data/nutch-dipe/1kcrawl/warc -dir /data/nutch-dipe/1kcrawl/segments/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NUTCH-2102) WARC Exporter
[ https://issues.apache.org/jira/browse/NUTCH-2102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14747300#comment-14747300 ] Julien Nioche commented on NUTCH-2102: -- The only modification to existing code is in the class 'src/plugin/protocol-http/src/java/org/apache/nutch/protocol/http/HttpResponse.java' where we added two new config elements : * store.http.request * store.http.headers which are used to keep the request and http headers verbatim in the content metadata. Both are set to false by default. Note that this is also used by [#55](https://github.com/apache/nutch/pull/55) > WARC Exporter > - > > Key: NUTCH-2102 > URL: https://issues.apache.org/jira/browse/NUTCH-2102 > Project: Nutch > Issue Type: Improvement > Components: commoncrawl, dumpers >Affects Versions: 1.10 >Reporter: Julien Nioche > Attachments: NUTCH-2102.patch > > > This patch adds a WARC exporter > [http://bibnum.bnf.fr/warc/WARC_ISO_28500_version1_latestdraft.pdf]. Unlike > the code submitted in [https://github.com/apache/nutch/pull/55] which is > based on the CommonCrawlDataDumper, this exporter is a MapReduce job and > hence should be able to cope with large segments in a timely fashion and also > is not limited to the local file system. > Later on we could have a WARCImporter to generate segments from WARC files, > which is outside the scope of the CCDD anyway. Also WARC is not specific to > CommonCrawl, which is why the package name does not reflect it. > I don't think it would be a problem to have both > [https://github.com/apache/nutch/pull/55] and this class providing similar > functionalities. > This class is called in the following way > ./nutch org.apache.nutch.tools.warc.WARCExporter > /data/nutch-dipe/1kcrawl/warc -dir /data/nutch-dipe/1kcrawl/segments/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
