[jira] [Commented] (NUTCH-2129) Track Protocol Status in Crawl Datum

2015-10-18 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14962594#comment-14962594
 ] 

Hudson commented on NUTCH-2129:
---

SUCCESS: Integrated in Nutch-trunk #3291 (See 
[https://builds.apache.org/job/Nutch-trunk/3291/])
Fix for NUTCH-2129 - Add protocol status tracking to crawl datum contributed by 
Michael Joyce  this closes #68. (mattmann: 
[http://svn.apache.org/viewvc/nutch/trunk/?view=rev=1709306])
* trunk/CHANGES.txt
* trunk/src/java/org/apache/nutch/metadata/Nutch.java
* 
trunk/src/plugin/lib-http/src/java/org/apache/nutch/protocol/http/api/HttpBase.java
* trunk/src/plugin/protocol-ftp/src/java/org/apache/nutch/protocol/ftp/Ftp.java


> Track Protocol Status in Crawl Datum
> 
>
> Key: NUTCH-2129
> URL: https://issues.apache.org/jira/browse/NUTCH-2129
> Project: Nutch
>  Issue Type: Improvement
>Affects Versions: 2.3, 1.10
>Reporter: Michael Joyce
>Assignee: Chris A. Mattmann
> Fix For: 2.4, 1.11
>
>
> It's become necessary on a few crawls that I run to get protocol status code 
> stats. After speaking with [~lewismc] it seemed that there might not be a 
> super convenient way of doing this as is, but it would be great to be able to 
> add the functionality necessary to pull this information out.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NUTCH-2129) Track Protocol Status in Crawl Datum

2015-10-18 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14962572#comment-14962572
 ] 

ASF GitHub Bot commented on NUTCH-2129:
---

Github user asfgit closed the pull request at:

https://github.com/apache/nutch/pull/68


> Track Protocol Status in Crawl Datum
> 
>
> Key: NUTCH-2129
> URL: https://issues.apache.org/jira/browse/NUTCH-2129
> Project: Nutch
>  Issue Type: Improvement
>Affects Versions: 2.3, 1.10
>Reporter: Michael Joyce
>Assignee: Chris A. Mattmann
> Fix For: 2.4, 1.11
>
>
> It's become necessary on a few crawls that I run to get protocol status code 
> stats. After speaking with [~lewismc] it seemed that there might not be a 
> super convenient way of doing this as is, but it would be great to be able to 
> add the functionality necessary to pull this information out.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NUTCH-2129) Track Protocol Status in Crawl Datum

2015-10-07 Thread Michael Joyce (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14947002#comment-14947002
 ] 

Michael Joyce commented on NUTCH-2129:
--

Fixed the unnecessary init that [~jnioche] caught. Thanks much for the reviews.

> Track Protocol Status in Crawl Datum
> 
>
> Key: NUTCH-2129
> URL: https://issues.apache.org/jira/browse/NUTCH-2129
> Project: Nutch
>  Issue Type: Improvement
>Affects Versions: 2.3, 1.10
>Reporter: Michael Joyce
> Fix For: 2.4, 1.11
>
>
> It's become necessary on a few crawls that I run to get protocol status code 
> stats. After speaking with [~lewismc] it seemed that there might not be a 
> super convenient way of doing this as is, but it would be great to be able to 
> add the functionality necessary to pull this information out.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NUTCH-2129) Track Protocol Status in Crawl Datum

2015-10-06 Thread Michael Joyce (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945766#comment-14945766
 ] 

Michael Joyce commented on NUTCH-2129:
--

Hey folks, updated PR with the metadata approach for HTTP and FTP. Let me know 
if you have any concerns.

> Track Protocol Status in Crawl Datum
> 
>
> Key: NUTCH-2129
> URL: https://issues.apache.org/jira/browse/NUTCH-2129
> Project: Nutch
>  Issue Type: Improvement
>Affects Versions: 2.3, 1.10
>Reporter: Michael Joyce
> Fix For: 2.4, 1.11
>
>
> It's become necessary on a few crawls that I run to get protocol status code 
> stats. After speaking with [~lewismc] it seemed that there might not be a 
> super convenient way of doing this as is, but it would be great to be able to 
> add the functionality necessary to pull this information out.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NUTCH-2129) Track Protocol Status in Crawl Datum

2015-10-01 Thread Michael Joyce (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14939939#comment-14939939
 ] 

Michael Joyce commented on NUTCH-2129:
--

Thanks Julien. I figured there would probably be a few thoughts on this, so I 
appreciate the feedback. I'll checkout the stuff you mentioned. Thanks for the 
ideas.

> Track Protocol Status in Crawl Datum
> 
>
> Key: NUTCH-2129
> URL: https://issues.apache.org/jira/browse/NUTCH-2129
> Project: Nutch
>  Issue Type: Improvement
>Affects Versions: 2.3, 1.10
>Reporter: Michael Joyce
> Fix For: 2.4, 1.11
>
>
> It's become necessary on a few crawls that I run to get protocol status code 
> stats. After speaking with [~lewismc] it seemed that there might not be a 
> super convenient way of doing this as is, but it would be great to be able to 
> add the functionality necessary to pull this information out.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NUTCH-2129) Track Protocol Status in Crawl Datum

2015-10-01 Thread Julien Nioche (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14939503#comment-14939503
 ] 

Julien Nioche commented on NUTCH-2129:
--

I'd rather keep it simple and not modify the CrawlDatum so much. Why don't you 
simply add a config element and optionally store the code in the metadata?
BTW we already have the option to store the response headers - see 
[https://github.com/apache/nutch/commit/23c7761aff830db82a1e44b84bf81265639c9a26].
 You could use that and simply reparse the first line to get the code.


> Track Protocol Status in Crawl Datum
> 
>
> Key: NUTCH-2129
> URL: https://issues.apache.org/jira/browse/NUTCH-2129
> Project: Nutch
>  Issue Type: Improvement
>Affects Versions: 2.3, 1.10
>Reporter: Michael Joyce
> Fix For: 2.4, 1.11
>
>
> It's become necessary on a few crawls that I run to get protocol status code 
> stats. After speaking with [~lewismc] it seemed that there might not be a 
> super convenient way of doing this as is, but it would be great to be able to 
> add the functionality necessary to pull this information out.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NUTCH-2129) Track Protocol Status in Crawl Datum

2015-09-30 Thread Michael Joyce (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14939124#comment-14939124
 ] 

Michael Joyce commented on NUTCH-2129:
--

Hi folks,

Initial pull request up to address this. Note that at the moment this only 
includes updates such that HTTP status codes are saved. I figured it would be 
best to get a conversation started on this before I dive into it too much since 
it's a rather core data structure.

Thoughts or ideas on this folks?

> Track Protocol Status in Crawl Datum
> 
>
> Key: NUTCH-2129
> URL: https://issues.apache.org/jira/browse/NUTCH-2129
> Project: Nutch
>  Issue Type: Improvement
>Affects Versions: 2.3, 1.10
>Reporter: Michael Joyce
> Fix For: 2.4, 1.11
>
>
> It's become necessary on a few crawls that I run to get protocol status code 
> stats. After speaking with [~lewismc] it seemed that there might not be a 
> super convenient way of doing this as is, but it would be great to be able to 
> add the functionality necessary to pull this information out.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NUTCH-2129) Track Protocol Status in Crawl Datum

2015-09-30 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/NUTCH-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14939122#comment-14939122
 ] 

ASF GitHub Bot commented on NUTCH-2129:
---

GitHub user MJJoyce opened a pull request:

https://github.com/apache/nutch/pull/68

NUTCH-2129 - Add protocol status tracking to crawl datum

- Add protocol status information to CrawlDatum.
- Bump CrawlDatum version number.
- Save HTTP Protocol status in CrawlDatum in lib-http.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/MJJoyce/nutch NUTCH-2129

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nutch/pull/68.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #68


commit d4c6d864752f17eeac0cb34032502e28743e3d17
Author: Michael Joyce 
Date:   2015-10-01T00:50:55Z

NUTCH-2129 - Add protocol status tracking to crawl datum

- Add protocol status information to CrawlDatum.
- Bump CrawlDatum version number.
- Save HTTP Protocol status in CrawlDatum in lib-http.




> Track Protocol Status in Crawl Datum
> 
>
> Key: NUTCH-2129
> URL: https://issues.apache.org/jira/browse/NUTCH-2129
> Project: Nutch
>  Issue Type: Improvement
>Affects Versions: 2.3, 1.10
>Reporter: Michael Joyce
> Fix For: 2.4, 1.11
>
>
> It's become necessary on a few crawls that I run to get protocol status code 
> stats. After speaking with [~lewismc] it seemed that there might not be a 
> super convenient way of doing this as is, but it would be great to be able to 
> add the functionality necessary to pull this information out.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)