[ https://issues.apache.org/jira/browse/IMPALA-11325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joe McDonnell resolved IMPALA-11325. ------------------------------------ Fix Version/s: Impala 4.2.0 Assignee: Joe McDonnell Resolution: Fixed > Impala-shell hits UnicodeDecodeError when outputting Unicode via --output_file > ------------------------------------------------------------------------------ > > Key: IMPALA-11325 > URL: https://issues.apache.org/jira/browse/IMPALA-11325 > Project: IMPALA > Issue Type: Bug > Components: Clients > Affects Versions: Impala 4.2.0 > Reporter: Joe McDonnell > Assignee: Joe McDonnell > Priority: Blocker > Fix For: Impala 4.2.0 > > > When running impala-shell and trying to output Unicode to a fail via --output > file, it fails: > {noformat} > ishell -B -q "select '引'" --output_file=joetest3.txt > /home/joe/view2/Impala/shell/option_parser.py:359: UnicodeWarning: Unicode > equal comparison failed to convert both arguments to Unicode - interpreting > them as being unequal > if '--live_progress' in sys.argv and '--disable_live_progress' in sys.argv: > /home/joe/view2/Impala/shell/option_parser.py:363: UnicodeWarning: Unicode > equal comparison failed to convert both arguments to Unicode - interpreting > them as being unequal > if '--strict_hs2_protocol' in sys.argv: > /home/joe/view2/Impala/shell/option_parser.py:369: UnicodeWarning: Unicode > equal comparison failed to convert both arguments to Unicode - interpreting > them as being unequal > if '--verbose' in sys.argv and '--quiet' in sys.argv: > Starting Impala Shell with no authentication using Python 2.7.16 > Warning: live_progress only applies to interactive shell sessions, and is > being skipped for now. > Opened TCP connection to localhost:21050 > Connected to localhost:21050 > Server version: impalad version 4.1.0-SNAPSHOT DEBUG (build > 4236c307b971881a3b1d85068db5b053a9c34cfa) > Query: select '引' > Query submitted at: 2022-05-31 08:31:50 (Coordinator: > http://joemcdonnell:25000) > Query progress can be monitored at: > http://joemcdonnell:25000/query_plan?query_id=2347462fe8a18544:bbeedc1800000000 > UnicodeDecodeError : 'ascii' codec can't decode byte 0xe5 in position 0: > ordinal not in range(128) > Please check for columns containing binary data to find the possible source > of the error. > Could not execute command: select '引'{noformat} > This is specific to file output. This same query works if outputting to the > console. > This line seems to be the problem: > {noformat} > with open(self.filename, 'ab') as out_file: > # Note that instances of this class do not persist, so it's fine to > # close the we close the file handle after each write. > out_file.write(formatted_data.encode('utf-8')) # file opened in > binary mode <-------- > out_file.write(b'\n') > {noformat} > [https://github.com/apache/impala/blob/master/shell/shell_output.py#L115] > It seems to work if we remove the .encode('utf-8'). -- This message was sent by Atlassian Jira (v8.20.10#820010)