[ 
https://issues.apache.org/jira/browse/HIVE-4199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617454#comment-13617454
 ] 

Phabricator commented on HIVE-4199:
-----------------------------------

omalley has commented on the revision "HIVE-4199 [jira] ORC writer doesn't 
handle non-UTF8 encoded Text properly".

INLINE COMMENTS
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/StringRedBlackTree.java:44 Let's 
just remove the current add method and move exclusively to Text for the 
StringRedBlackTree. I should have known better than using String internally.
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java:717 We're going 
to have the same problem here with the statistics. If we need to support 
non-utf8, we need to update the statistics to use Text also.

REVISION DETAIL
  https://reviews.facebook.net/D9501

To: kevinwilfong, sxyuan
Cc: JIRA, omalley

                
> ORC writer doesn't handle non-UTF8 encoded Text properly
> --------------------------------------------------------
>
>                 Key: HIVE-4199
>                 URL: https://issues.apache.org/jira/browse/HIVE-4199
>             Project: Hive
>          Issue Type: Bug
>          Components: Serializers/Deserializers
>            Reporter: Samuel Yuan
>            Assignee: Samuel Yuan
>            Priority: Minor
>         Attachments: HIVE-4199.HIVE-4199.HIVE-4199.D9501.1.patch, 
> HIVE-4199.HIVE-4199.HIVE-4199.D9501.2.patch, 
> HIVE-4199.HIVE-4199.HIVE-4199.D9501.3.patch, 
> HIVE-4199.HIVE-4199.HIVE-4199.D9501.4.patch
>
>
> StringTreeWriter currently converts fields stored as Text objects into 
> Strings. This can lose information (see 
> http://en.wikipedia.org/wiki/Replacement_character#Replacement_character), 
> and is also unnecessary since the dictionary stores Text objects.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Reply via email to