[jira] [Commented] (NUTCH-1949) Dump out the Nuth data into the Common Crawl format
[
https://issues.apache.org/jira/browse/NUTCH-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346364#comment-14346364
]
Chris A. Mattmann commented on NUTCH-1949:
--
+1
> Dump out the Nuth data into the Common Crawl format
> ---
>
> Key: NUTCH-1949
> URL: https://issues.apache.org/jira/browse/NUTCH-1949
> Project: Nutch
> Issue Type: New Feature
>Reporter: Giuseppe Totaro
>Assignee: Giuseppe Totaro
> Attachments: CommonCrawlDataDumper.pdf, CommonCrawlDataDumper.xlsx,
> CommonCrawlDataDumper_v02.pdf
>
>
> We are going to develop a {{CommonCrawlDataDumper.java}} class. The
> {{CommonCrawlDataDumper}} is a tool able to perfom the following steps:
> # deserialize the crawled data from Nutch
> # map serialized data on the proper JSON structure
> # serialize the data into [CBOR|http://cbor.io] format
> # optionally, compress the serialized data using {{gzip}}
> This tool has to be able to work with either single Nutch segments or
> directory including segments as input data.
> Thanks [~lewismc] and [~chrismattmann] for your great suggestions, support
> and code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NUTCH-1949) Dump out the Nuth data into the Common Crawl format
[
https://issues.apache.org/jira/browse/NUTCH-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346265#comment-14346265
]
Jorge Luis Betancourt Gonzalez commented on NUTCH-1949:
---
+1
> Dump out the Nuth data into the Common Crawl format
> ---
>
> Key: NUTCH-1949
> URL: https://issues.apache.org/jira/browse/NUTCH-1949
> Project: Nutch
> Issue Type: New Feature
>Reporter: Giuseppe Totaro
>Assignee: Giuseppe Totaro
> Attachments: CommonCrawlDataDumper.pdf, CommonCrawlDataDumper.xlsx,
> CommonCrawlDataDumper_v02.pdf
>
>
> We are going to develop a {{CommonCrawlDataDumper.java}} class. The
> {{CommonCrawlDataDumper}} is a tool able to perfom the following steps:
> # deserialize the crawled data from Nutch
> # map serialized data on the proper JSON structure
> # serialize the data into [CBOR|http://cbor.io] format
> # optionally, compress the serialized data using {{gzip}}
> This tool has to be able to work with either single Nutch segments or
> directory including segments as input data.
> Thanks [~lewismc] and [~chrismattmann] for your great suggestions, support
> and code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NUTCH-1949) Dump out the Nuth data into the Common Crawl format
[
https://issues.apache.org/jira/browse/NUTCH-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346258#comment-14346258
]
Lewis John McGibbney commented on NUTCH-1949:
-
Review undertaken by [~jnioche], [~chrismattmann] and [~lewismc] on this patch.
There is a roadmap to make this an indexing plugin. I will commit EoB tomorrow
unless objections and we can open another issue to get it ported to an indexing
plugin.
> Dump out the Nuth data into the Common Crawl format
> ---
>
> Key: NUTCH-1949
> URL: https://issues.apache.org/jira/browse/NUTCH-1949
> Project: Nutch
> Issue Type: New Feature
>Reporter: Giuseppe Totaro
>Assignee: Giuseppe Totaro
> Attachments: CommonCrawlDataDumper.pdf, CommonCrawlDataDumper.xlsx,
> CommonCrawlDataDumper_v02.pdf
>
>
> We are going to develop a {{CommonCrawlDataDumper.java}} class. The
> {{CommonCrawlDataDumper}} is a tool able to perfom the following steps:
> # deserialize the crawled data from Nutch
> # map serialized data on the proper JSON structure
> # serialize the data into [CBOR|http://cbor.io] format
> # optionally, compress the serialized data using {{gzip}}
> This tool has to be able to work with either single Nutch segments or
> directory including segments as input data.
> Thanks [~lewismc] and [~chrismattmann] for your great suggestions, support
> and code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NUTCH-1949) Dump out the Nuth data into the Common Crawl format
[
https://issues.apache.org/jira/browse/NUTCH-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343685#comment-14343685
]
Giuseppe Totaro commented on NUTCH-1949:
Hi [~lewismc], you can review the code
[here|https://reviews.apache.org/r/31579/].
> Dump out the Nuth data into the Common Crawl format
> ---
>
> Key: NUTCH-1949
> URL: https://issues.apache.org/jira/browse/NUTCH-1949
> Project: Nutch
> Issue Type: New Feature
>Reporter: Giuseppe Totaro
>Assignee: Giuseppe Totaro
> Attachments: CommonCrawlDataDumper.pdf, CommonCrawlDataDumper.xlsx,
> CommonCrawlDataDumper_v02.pdf
>
>
> We are going to develop a {{CommonCrawlDataDumper.java}} class. The
> {{CommonCrawlDataDumper}} is a tool able to perfom the following steps:
> # deserialize the crawled data from Nutch
> # map serialized data on the proper JSON structure
> # serialize the data into [CBOR|http://cbor.io] format
> # optionally, compress the serialized data using {{gzip}}
> This tool has to be able to work with either single Nutch segments or
> directory including segments as input data.
> Thanks [~lewismc] and [~chrismattmann] for your great suggestions, support
> and code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NUTCH-1949) Dump out the Nuth data into the Common Crawl format
[
https://issues.apache.org/jira/browse/NUTCH-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343676#comment-14343676
]
Lewis John McGibbney commented on NUTCH-1949:
-
[~gostep] do you have a link to the patch?
> Dump out the Nuth data into the Common Crawl format
> ---
>
> Key: NUTCH-1949
> URL: https://issues.apache.org/jira/browse/NUTCH-1949
> Project: Nutch
> Issue Type: New Feature
>Reporter: Giuseppe Totaro
>Assignee: Giuseppe Totaro
> Attachments: CommonCrawlDataDumper.pdf, CommonCrawlDataDumper.xlsx,
> CommonCrawlDataDumper_v02.pdf
>
>
> We are going to develop a {{CommonCrawlDataDumper.java}} class. The
> {{CommonCrawlDataDumper}} is a tool able to perfom the following steps:
> # deserialize the crawled data from Nutch
> # map serialized data on the proper JSON structure
> # serialize the data into [CBOR|http://cbor.io] format
> # optionally, compress the serialized data using {{gzip}}
> This tool has to be able to work with either single Nutch segments or
> directory including segments as input data.
> Thanks [~lewismc] and [~chrismattmann] for your great suggestions, support
> and code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NUTCH-1949) Dump out the Nuth data into the Common Crawl format
[
https://issues.apache.org/jira/browse/NUTCH-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14341410#comment-14341410
]
Lewis John McGibbney commented on NUTCH-1949:
-
Review shortly Giuseppe this is grand
On Friday, February 27, 2015, Giuseppe Totaro (JIRA)
--
*Lewis*
> Dump out the Nuth data into the Common Crawl format
> ---
>
> Key: NUTCH-1949
> URL: https://issues.apache.org/jira/browse/NUTCH-1949
> Project: Nutch
> Issue Type: New Feature
>Reporter: Giuseppe Totaro
>Assignee: Giuseppe Totaro
> Attachments: CommonCrawlDataDumper.pdf, CommonCrawlDataDumper.xlsx,
> CommonCrawlDataDumper_v02.pdf
>
>
> We are going to develop a {{CommonCrawlDataDumper.java}} class. The
> {{CommonCrawlDataDumper}} is a tool able to perfom the following steps:
> # deserialize the crawled data from Nutch
> # map serialized data on the proper JSON structure
> # serialize the data into [CBOR|http://cbor.io] format
> # optionally, compress the serialized data using {{gzip}}
> This tool has to be able to work with either single Nutch segments or
> directory including segments as input data.
> Thanks [~lewismc] and [~chrismattmann] for your great suggestions, support
> and code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NUTCH-1949) Dump out the Nuth data into the Common Crawl format
[
https://issues.apache.org/jira/browse/NUTCH-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14336558#comment-14336558
]
Chris A. Mattmann commented on NUTCH-1949:
--
+1 Julien that's a great idea. Then we can run ./bin/nutch index and instead of
only going to Solr or ElasticSearch or whatever, this would also (or instead
of) generate CommonCrawl data (an "index" of sorts). Let's explore this
[~gostep] after you get the initial code whipped up. [~gostep] can you also
share the arch diagram you have been working on?
> Dump out the Nuth data into the Common Crawl format
> ---
>
> Key: NUTCH-1949
> URL: https://issues.apache.org/jira/browse/NUTCH-1949
> Project: Nutch
> Issue Type: New Feature
>Reporter: Giuseppe Totaro
>Assignee: Giuseppe Totaro
>
> We are going to develop a {{CommonCrawlDataDumper.java}} class. The
> {{CommonCrawlDataDumper}} is a tool able to perfom the following steps:
> # deserialize the crawled data from Nutch
> # map serialized data on the proper JSON structure
> # serialize the data into [CBOR|http://cbor.io] format
> # optionally, compress the serialized data using {{gzip}}
> This tool has to be able to work with either single Nutch segments or
> directory including segments as input data.
> Thanks [~lewismc] and [~chrismattmann] for your great suggestions, support
> and code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NUTCH-1949) Dump out the Nuth data into the Common Crawl format
[
https://issues.apache.org/jira/browse/NUTCH-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14336272#comment-14336272
]
Julien Nioche commented on NUTCH-1949:
--
[~gostep] maybe see if this can be done as a plugin by extending the
org.apache.nutch.indexer.IndexWriter endpoint. This way the content will be
generated by using the generic index command.
> Dump out the Nuth data into the Common Crawl format
> ---
>
> Key: NUTCH-1949
> URL: https://issues.apache.org/jira/browse/NUTCH-1949
> Project: Nutch
> Issue Type: New Feature
>Reporter: Giuseppe Totaro
>Assignee: Giuseppe Totaro
>
> We are going to develop a {{CommonCrawlDataDumper.java}} class. The
> {{CommonCrawlDataDumper}} is a tool able to perfom the following steps:
> # deserialize the crawled data from Nutch
> # map serialized data on the proper JSON structure
> # serialize the data into [CBOR|http://cbor.io] format
> # optionally, compress the serialized data using {{gzip}}
> This tool has to be able to work with either single Nutch segments or
> directory including segments as input data.
> Thanks [~lewismc] and [~chrismattmann] for your great suggestions, support
> and code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NUTCH-1949) Dump out the Nuth data into the Common Crawl format
[
https://issues.apache.org/jira/browse/NUTCH-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14335058#comment-14335058
]
Lewis John McGibbney commented on NUTCH-1949:
-
Thanks for logging this [~gostep], we will see if the tool is generic and
appealing enough for integration in to Nutch, my hope is that it will be :)
> Dump out the Nuth data into the Common Crawl format
> ---
>
> Key: NUTCH-1949
> URL: https://issues.apache.org/jira/browse/NUTCH-1949
> Project: Nutch
> Issue Type: New Feature
>Reporter: Giuseppe Totaro
>
> We are going to develop a {{CommonCrawlDataDumper.java}} class. The
> {{CommonCrawlDataDumper}} is a tool able to perfom the following steps:
> # deserialize the crawled data from Nutch
> # map serialized data on the proper JSON structure
> # serialize the data into [CBOR|http://cbor.io] format
> # optionally, compress the serialized data using {{gzip}}
> This tool has to be able to work with either single Nutch segments or
> directory including segments as input data.
> Thanks [~lewismc] and [~chrismattmann] for your great suggestions, support
> and code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
