[
https://issues.apache.org/jira/browse/HIVE-5263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13764631#comment-13764631
]
Mohammad Kamrul Islam commented on HIVE-5263:
---------------------------------------------
Very good work Brock.
I did some work on this. Let me share with you.
For clonePlan(), I first planned to use kayo.copy(). there is an improvement
that Leo mentioned in my question at kryo-user:
https://groups.google.com/forum/#!topic/kryo-users/vB5sHGCogKQ
I think copy would be better than serialized and deserilized. There is a
related Hive jira:
https://issues.apache.org/jira/browse/HIVE-4396
I then tried with serialization and deserialization approach.
I created a new kryo object just to mask the non-transient field for normal
processing. Some code snippets:
private static ThreadLocal<Kryo> modifuedKryo = new ThreadLocal<Kryo>() {
@Override
protected synchronized Kryo initialValue() {
Kryo kryo = new Kryo();
kryo.setClassLoader(Thread.currentThread().getContextClassLoader());
kryo.register(java.sql.Date.class, new SqlDateSerializer());
removeField(kryo, ColumnInfo.class, "objectInspector");
removeField(kryo, Operator.class, "colExprMap");
removeField(kryo, MapWork.class, "opParseCtxMap");
removeField(kryo, MapWork.class, "joinTree");
return kryo;
};
protected void removeField(Kryo kryo, Class type, String fieldName) {
FieldSerializer fld = new FieldSerializer(kryo, type);
fld.removeField(fieldName);
kryo.register(type, fld);
}
};
There are some discussion in kayo-user:
https://groups.google.com/forum/#!topic/kryo-users/BDIBMqaAIyY
> Query Plan cloning time could be improved by using Kryo
> --------------------------------------------------------
>
> Key: HIVE-5263
> URL: https://issues.apache.org/jira/browse/HIVE-5263
> Project: Hive
> Issue Type: Improvement
> Reporter: Brock Noland
> Assignee: Brock Noland
> Attachments: HIVE-5263.patch, HIVE-5263.patch
>
>
> In HIVE-1511 we implemented Kryo for serialization but did not use for query
> plan cloning. As was discussed there it's a possible speed improvement. In
> addition the current XML serialization method does not work with Java 7 so
> using Kryo is a workaround for this issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
