for tranform,let the user specify the serde
-------------------------------------------
Key: HIVE-708
URL: https://issues.apache.org/jira/browse/HIVE-708
Project: Hadoop Hive
Issue Type: New Feature
Components: Serializers/Deserializers
Reporter: Namit Jain
Assignee: Namit Jain
Currently, LazySimpleSerDe is used to send data to the user transformation
functions - it would be useful to let the user specify the format of the data.
Specifically, it would be very easy and useful to accommodate:
(cut and paste from Venky's mail)
Here's the typedbytes stuff that Dumbo uses.
http://issues.apache.org/jira/browse/HADOOP-1722
From:
http://static.last.fm/johan/huguk-20090414/klaas-hadoop-1722.pdf
Timings for IP count program on 300gigs of weblogs:
Java: 8 minutes
Dumbo with typed bytes: 10 minutes
Hive: 13 minutes
Dumbo without typed bytes: 16 minutes
They also have a fast python decoder for this, which is apparently 25% faster
than the python version.
http://github.com/klbostee/ctypedbytes/tree/master
http://dumbotics.com/2009/05/31/dumbo-on-clouderas-distribution/
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.