[ 
https://issues.apache.org/jira/browse/TIKA-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15401715#comment-15401715
 ] 

Shabanali Faghani edited comment on TIKA-2038 at 8/1/16 8:45 AM:
-----------------------------------------------------------------

OK, so to give more details about my library to this community and also in 
response to your concerns I would to say:

1) You are right, my repo on github is fairly new (less than 1 year) but its 
algorithm is not new. I developed this library 4 years ago in order to be used 
in a large-scale project… and it works well from that time till now. It was 
under a load of ~1.2 billion pages in the peak time. The bug that I’ve fixed in 
last week was just a tiny mistake that occurred during refactoring the code 
before the first release.

2) Since the accuracy was much more important than performance for us, I 
haven’t done a thorough performance test. Nevertheless, bellow I’ve provided 
the results of a small test that was done on my laptop (intel core i3, java 6, 
Xmx: default (don’t care)):
||Subdirectory||#docs||Total Size (KB)||Average Size (KB)||Detection Time 
(millisecond)||Average Time (millisecond)||
|UTF-8|657|32,216|49|26658|40|
|Windows-1251|314|30,941|99|4423|14|
|GBK|419|43,374|104|20317|48|
|Windows-1256|645|66,592|103|9451|14|
|Shift_JIS|640|25,973|41|7617|11|

Let’s have a little bit more precise look at these results. Due to the logic of 
my algorithm, for the first row of this table, i.e UTF-8, only Mozilla JCharDet 
was used (no JSoup and no ICU4J). But as you can see from this table the 
required time is greater than the three other cases for which documents were 
parsed using JSoup and also the both JCharDet and ICU4J were involved in 
detection prrocess. It means that if the encoding of a page is UTF-8 the 
required time for a positive response from Mozilla JChardet is often greater 
than the required time for … 
* get a negative response from Mozilla JChardet +
* decode the input byte array using “ISO-8859-1”+
* parse that doc and creating DOM tree +
* extract text from DOM tree +
* encode the extracted text using “ISO-8859-1” +
* and detecting its encoding by using icu4j  
… when the encoding of a page is not UTF-8!! In brief, 40 > 14, 11 , … in the 
above table. 

Now let’s have a look at the [distribution of character encodings for websites| 
https://w3techs.com/technologies/history_overview/character_encoding]. Since 
~87% of all websites use UTF-8, if we use a statistical computation for 
computing the weighted average time for detecting encoding of a custom html 
document, I think we would get a similar estimate for both IUST-HTMLCharDet and 
Tika-EncodingDetector. Because, this estimate is strongly biased by Mozilla 
JCharDet and as we know this tool is used in the both algorithms in a similar 
way. Nevertheless, for performance optimizations I will do some tests for …
* using a Regex instead of navigating in DOM tree for seeking charsets in Meta 
tags
* stripping HTML Markups, Scripts and embedded CSSs directly instead of using a 
html parser

3) For computing the accuracy of Tika's legacy method I’ve provided a comment 
below your current evaluation results. As I’ve explained there, the results of 
your current evaluation couldn’t be compared with my evaluation.

bq. Perhaps we could add some code to do that?
Of course, but from experience when I use open sources in my projects, due to 
the versioning and updating considerations I don’t shift my code into them 
unless there wouldn’t be any other suitable solution/option. But pulling a part 
of their code into my projects is *another story*! :)


was (Author: faghani):
OK, so to give more details about my library to this community and also in 
response to your concerns I would to say:

1) You are right, my repo on github is fairly new (less than 1 year) but its 
algorithm is not new. I developed this library 4 years ago in order to be used 
in a large-scale project… and it works well from that time till now. It was 
under a load of ~1.2 billion pages in the peak time. The bug that I’ve fixed in 
last week was just a tiny mistake that occurred during refactoring the code 
before the first release.

2) Since the accuracy was much more important than performance for us, I 
haven’t done a thorough performance test. Nevertheless, bellow I’ve provided 
the results of a small test that was done on my laptop (intel core i3, java 6, 
Xmx: default (don’t care)):
||Subdirectory||#docs||Total Size (KB)||Average Size (KB)||Detection Time 
(millisecond)||Average Time (millisecond)||
|UTF-8|657|32,216|49|26658|40|
|Windows-1251|314|30,941|99|4423|14|
|GBK|419|43,374|104|20317|48|
|Windows-1256|645|66,592|103|9451|14|
|Shift_JIS|640|25,973|41|7617|11|

Let’s have a little bit more precise look at these results. Due to the logic of 
my algorithm, for the first row of this table, i.e UTF-8, only Mozilla JCharDet 
was used (no JSoup and no ICU4J). But as you can see from this table the 
required time is greater than the three other cases for which documents were 
parsed using JSoup and also the both JCharDet and ICU4J were involved in 
detection prrocess. It means that if the encoding of a page is UTF-8 the 
required time for a positive response from Mozilla JChardet is often greater 
than the required time for … 
* get a negative response from Mozilla JChardet +
* decode the input byte array using “ISO-8859-1”+
* parse that doc and creating DOM tree +
* extract text from DOM tree +
* encode the extracted text using “ISO-8859-1” +
* and detecting its encoding by using icu4j  
… when the encoding of a page is not UTF-8!! In brief, 40 > 14, 11 , … in the 
above table. 

Now let’s have a look at the [distribution of character encodings for websites| 
https://w3techs.com/technologies/history_overview/character_encoding]. Since 
~87% of all websites use UTF-8, if we use a statistical computation for 
computing the weighted average time for detecting encoding of a custom html 
document, I think we would get a similar estimate for both IUST-HTMLCharDet and 
Tika-EncodingDetector. Because, this estimate is strongly biased by Mozilla 
JCharDet and as we know this tool is used in the both algorithms in a similar 
way. Nevertheless, for performance optimizations I will do some tests for …
* using a Regex instead of navigating in DOM tree for seeking charsets in Meta 
tags
* stripping HTML Markups, Scripts and embedded CSSs directly instead of using a 
html parser

3) For computing the accuracy of Tika's legacy method I’ve provided a comment 
below your current evaluation results. As I’ve explained there, the results of 
your current evaluation couldn’t be compared with my evaluation.

bq. Perhaps we could add some code to do that?
Of course, but from experience when I use open sources in my projects, due to 
the versioning and updating considerations I don’t shift my code into them 
unless there wouldn’t be any other suitable solution/option. But pulling other 
code into my projects is *another story*! :)

> A more accurate facility for detecting Charset Encoding of HTML documents
> -------------------------------------------------------------------------
>
>                 Key: TIKA-2038
>                 URL: https://issues.apache.org/jira/browse/TIKA-2038
>             Project: Tika
>          Issue Type: Improvement
>          Components: core, detector
>            Reporter: Shabanali Faghani
>            Priority: Minor
>         Attachments: tika_1_14-SNAPSHOT_encoding_detector.zip
>
>
> Currently, Tika uses icu4j for detecting charset encoding of HTML documents 
> as well as the other naturally text documents. But the accuracy of encoding 
> detector tools, including icu4j, in dealing with the HTML documents is 
> meaningfully less than from which the other text documents. Hence, in our 
> project I developed a library that works pretty well for HTML documents, 
> which is available here: https://github.com/shabanali-faghani/IUST-HTMLCharDet
> Since Tika is widely used with and within some of other Apache stuffs such as 
> Nutch, Lucene, Solr, etc. and these projects are strongly in connection with 
> the HTML documents, it seems that having such an facility in Tika also will 
> help them to become more accurate.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to