[ http://issues.apache.org/jira/browse/HADOOP-136?page=comments#action_12378549 ]
Michel Tourn commented on HADOOP-136: ------------------------------------- OK, given the need for 100% backward-compatibility and given that I will only use this format internally: I will go with Sameer's suggestion: have a separate class org.apache.hadoop.io.LargeUTF8.java I will just make its length field 4-bytes, rather than var-length: Otherwise this would complicate things like the ad-hoc offset computations in UTF8.Comparator. Should I make a TestLargeUTF8 based on TestUTF8? --- Concerning 4-bytes-long UTF-8 characters: it seems that this situation does not occur in "Java-modified-UTF8" The 4-byte chars are represented as 3+3. See Modified UTF-8 in: http://java.sun.com/developer/technicalArticles/Intl/Supplementary/ > Overlong UTF8's not handled well > -------------------------------- > > Key: HADOOP-136 > URL: http://issues.apache.org/jira/browse/HADOOP-136 > Project: Hadoop > Type: Bug > Components: io > Reporter: Dick King > Priority: Minor > > When we feed an overlong string to the UTF8 constructor, two suboptimal > things happen. > First, we truncate to 0xffff/3 characters on the assumption that every > character takes three bytes in UTF8. This can truncate strings that don't > need it, and it can be overoptimistic since there are characters that render > as four bytes in UTF8. > Second, the code doesn't actually handle four-byte characters. > Third, there's a behavioral discontinuity. If the string is "discovered" to > be overlong by the arbitrary limit described above, we truncate with a log > message, otherwise we signal a RuntimeException. One feels that both forms > of truncation should be treated alike. However, this issue is concealed by > the second issue; the exception will never be thrown because UTF8.utf8Length > can't return more than three times the length of its input. > I would recommend changing UTF8.utf8Length to let its caller know how many > characters of the input string will actually fit if there's an overflow > [perhaps by returning the negative of that number] and doing the truncation > accurately as needed. > -dk -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
