Hi Paul, I haven't used special characters in Flume, but I had previous issues in Java with Characters encoding, and they were solved by specifying the JVM default Character encoding, with: "-Dfile.encoding=UTF-8" (here for UTF-8)
Might be worth trying to add that in the Flume command line options? Or maybe on the front application ? Regards Jeff From: Paul Chavez [mailto:[email protected]] Sent: mardi 9 décembre 2014 20:25 To: [email protected] Subject: UTF-8 data mangled in flight Hello, Hoping to get some insight on where to further troubleshoot this issue. The scenario is we have a web application which accepts URL encoded UTF-8 characters (Cyrillic text in this instance) and then our web application sends this data to a Flume agent via HTTPSource with the JSONHandler. This agent then in turn sends the event along via Avro sink to another Flume agent which writes it to HDFS using the HDFS sink. We initially noticed the data was no longer valid in the HDFS file and after investigating have found the following: - The initial POST is correct, verified via a network trace and looking at binary data on the wire. - The Avro event sent from the Flume agent is mangled, again verified via network trace and looking at the binary payload. We do not explicitly set the content type header on the POST from our application as documentation states if not set then UTF-8 will be assumed. Can anyone elaborate on when/why this data is being corrupted? Thanks, Paul Chavez ________________________________ This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy. ______________________________________________________________________________________ www.accenture.com
