David Gao created IO-331: ---------------------------- Summary: BOMInputStream wrongly detects UTF-32LE_BOM files as UTF-16LE_BOM files in method getBOM() Key: IO-331 URL: https://issues.apache.org/jira/browse/IO-331 Project: Commons IO Issue Type: Bug Components: Streams/Writers Affects Versions: 2.3 Environment: OS: Win 7 x64 JDK: 1.7.03 Reporter: David Gao
Hi, The BOMInputStream works great for most UTF encoded files when detecting Byte Order Marks. However, if a file is UTF-32LE encoded with BOM the class takes it as UTF-16LE instead. This is not expected behavior. The problem comes from method getBOM(). And the first two bytes for UTF-16LE and UTF-32LE are the same, which might be the root cause of the problem. The following lists the bytes for UTF encodings for reference. The content is a BOM followed by letter 't'. ||Encoding||Byte 1||Byte 2||Byte 3||Byte 4|| || || || |UTF8|EF|BB|BF|74| | | | |UTF16-LE|FF|FE|74|00| | | | |UTF16-BE|FE|FF|00|74| | | | |UTF32-LE|FF|FE|00|00|74|00|00|00 |UTF32-BE|00|00|FE|FF|00|00|00|74 I personally used the following code to work around this problem at the moment. Hope it helps. {code} private void detectBOM(InputStream in) throws IOException{ List<ByteOrderMark> all=availableBOMs(); int max=0; for (ByteOrderMark bom : all) { max = Math.max(max, bom.length()); } byte[] firstBytes=new byte[max]; for (int i = 0; i < max; i++) { firstBytes[i]=(byte) in.read(); System.out.print(Integer.toHexString(firstBytes[i] & 0xff).toUpperCase()+" "); } boolean found=false; for (int j = max; j >1; j--) { byte[] _copy=Arrays.copyOf(firstBytes, j); for (ByteOrderMark mark : all) { found=Arrays.equals(_copy, mark.getBytes()); if (found) { System.out.println("\nBOM is: "+mark.getCharsetName()); break; } } if (found) break; } } private static List<ByteOrderMark> availableBOMs(){ List<ByteOrderMark> all=new ArrayList<ByteOrderMark>(); all.add(ByteOrderMark.UTF_8); all.add(ByteOrderMark.UTF_16BE); all.add(ByteOrderMark.UTF_16LE); all.add(ByteOrderMark.UTF_32BE); all.add(ByteOrderMark.UTF_32LE); return all; } {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira