[
https://issues.apache.org/jira/browse/HIVE-266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Shao updated HIVE-266:
----------------------------
Attachment: HIVE-266.1.patch
First cut.
I mainly followed Choice 1 while there are some changes in the details:
* TypeInfo: Added PrimitiveTypeEntry in TypeInfoUtils.java to describe a
primitive type: name, java primitive type, java primitive Class, WritableClass.
** Modified all codes to use TypeInfo to identify a type (instead of Class<?> -
we should never use Class<?> now)
* ObjectInspector: Add methods in PrimitiveObjectInspector to extract the Java
Object and the Writable Object
** In this way, we still allow both Text/String to flow in the system.
** They will have the same TypeInfo, but different PrimitiveObjectInspector:
JavaPrimitiveObjectInspector for Java Object and
StandardPrimitiveObjectInspector for Writable Object.
* SerDe:
** MetadataTypedColumnsetSerDe and DynamicSerDe will produce Java Objects, and
they use JavaPrimitiveObjectInspector as a foundation to create their
ObjectInspectors
** LazySimpleSerDe will produce Writable Objects, and it uses
StandardPrimitiveObjectInspector as a foundation to create its ObjectInspector
** All SerDes accepts Java Objects and Writable Objects in the serialization
path - that difference is handled by PrimitiveObjectInspectors
* Operators
** GroupByOperator and JoinOperator now do copy when caching Objects.
> Improve SerDe performance by using Text instead of String
> ---------------------------------------------------------
>
> Key: HIVE-266
> URL: https://issues.apache.org/jira/browse/HIVE-266
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
> Affects Versions: 0.2.0
> Reporter: Zheng Shao
> Priority: Critical
> Attachments: HIVE-266.1.patch
>
>
> A recent performance study showed that 2 places in Hive code has exhibited
> large cpu usage percentage:
> 1. String.getBytes() (UTF-8 encoding)
> 2. String.split()
> We should replace String with Text object to:
> 1. Avoid UTF-8 decoding and encoding
> 2. Reuse the Text object and avoid creating new objects for each column in
> each row like in String.split()
> This is expected to give a big (20%+) performance improvement to Hive.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
