[
https://issues.apache.org/jira/browse/HIVE-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12694315#action_12694315
]
Zheng Shao commented on HIVE-266:
---------------------------------
We had a discussion yesterday:
* Logical Data Types
** Used for defining table columns
** Example: String, Integer
* Physical Data Types
** Used for Internal Processing
** Examples: String/Text, Integer/IntWritable
* Efficiency
** Text is reusable
** Text is UTF-8
** Usually the same as serialized format
* Standard:
** Text is Hadoop standard
** String is Java standard
** Will be useful to interface with Thrift, ProtocolBuffer etc
* Now:
** Everything is String (and Integer)
* Future:
** Choice 1: Use only Text (and IntWritable)
*** How to interface with Thrift and other standard Java libraries?
** Choice 2: Use a combination of String/Text (and Integer/IntWritable)
*** Ideal case
*** Much more complicated: PhysicalDataTypes will change after SerDe
(ReduceSinkOperator or FileSinkOperator)
The conclusion is that we will go Choice 1: Replace String/Integer etc in Hive
with Text/IntWritable.
As a side-effect, all SerDe/UDF(including UDAF) needs to use Writables instead
of Java Primitive classes.
We will change hive internal SerDe/UDF functions. For user-defined SerDe/UDFs,
users will need to change them to make them work with Hive 0.4.
> Improve SerDe performance by using Text instead of String
> ---------------------------------------------------------
>
> Key: HIVE-266
> URL: https://issues.apache.org/jira/browse/HIVE-266
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
> Affects Versions: 0.2.0
> Reporter: Zheng Shao
> Priority: Critical
>
> A recent performance study showed that 2 places in Hive code has exhibited
> large cpu usage percentage:
> 1. String.getBytes() (UTF-8 encoding)
> 2. String.split()
> We should replace String with Text object to:
> 1. Avoid UTF-8 decoding and encoding
> 2. Reuse the Text object and avoid creating new objects for each column in
> each row like in String.split()
> This is expected to give a big (20%+) performance improvement to Hive.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
