[
https://issues.apache.org/jira/browse/HCATALOG-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205293#comment-13205293
]
[email protected] commented on HCATALOG-241:
--------------------------------------------------------
bq. On 2012-02-09 19:26:50, Francis Liu wrote:
bq. > /trunk/src/java/org/apache/hcatalog/data/HCatRecordSerDe.java, line 197
bq. > <https://reviews.apache.org/r/3819/diff/1/?file=73307#file73307line197>
bq. >
bq. > shouldn't this be called deserializeField?
bq.
bq. Sushanth Sowmyan wrote:
bq. Technically, for hive, no. :)
bq.
bq. Hive sits on top of the SerDe, and is using this SerDe to "serialize"
an object/objectinspector it provides to a Writable representation, which in
this case, is HCatRecord. So the process of converting an
object-objectinspector pair into HCatRecord, from hive's perspective, is
serialization.
bq.
bq. The terminology is a bit confusing depending on which layer we see it
from, but I've tried to keep it consistent with the other serdes, and name it
as hive would see it.
bq.
bq. Francis Liu wrote:
bq. Hmmm this is what I thought it was:
bq.
bq. Serialize: Hive/HCatRecord Object -> Storage System compatible writable
bq. Deserialize: Writable -> HCatRecord/Hive Object
bq.
bq. In the output side for HCat I use serDe.serialize on HCatRecord.
You aren't wrong - serialize is called when Hive "writes", and deserialize is
called when Hive "reads". Serialize takes an (o,oi) pair and returns a
Writable, and Deserialize takes a Writable and returns an object which is
readable using the appropriate OI.
It's just that in the context of HCatRecordSerDe, the Writable form here is a
HCatRecord as well. So, if we start with a hive (o,oi) pair, serializing that
results in a HCatRecord -- thus, this is a "serializeField" operation.
Deserializing a HCatRecord is a trivial do-nothing operation that returns that
HCatRecord itself, and also returns a HCatRecordObjectInspector on
.getObjectInspector() being called on it, thus allowing hive to crawl through a
HCatRecord.
And when this function is called from LazyHCatRecord, it's purpose is to
convert an underlying (o,oi) pair to a java object inside HCatRecord - an
operation we think of as deserialization because, to us, it's taking an
"underlying" format, and attempting to convert to a format we are used to.
However, that is still a serialize operation on the SerDe itself, since it uses
an established (o,oi) pair to return a HCatRecord entry.
bq. On 2012-02-09 19:26:50, Francis Liu wrote:
bq. > /trunk/src/java/org/apache/hcatalog/data/HCatRecord.java, line 143
bq. > <https://reviews.apache.org/r/3819/diff/1/?file=73305#file73305line143>
bq. >
bq. > you be comparing this. you're breaking the transitivity contract for
the compareTo method
bq.
bq. Francis Liu wrote:
bq. *you shouldn't be comparing this.
bq.
bq. Francis Liu wrote:
bq. oops i meant reflexive.
bq.
bq. Sushanth Sowmyan wrote:
bq. I'm not sure I understand - compareTo returns -1, 0 or 1 depending on
the comparison.
bq.
bq. a.compareTo(b) is intended to be negative of b.compareTo(a) unless
they are equal. A compare operation is not reflexive.
bq.
bq. Francis Liu wrote:
bq. When they are equal it should be reflexive. When they are not one
should be the inverse of the other.
bq.
bq. Ok the first statement sounds a bit contradicting.
bq.
bq. Formally what I mean to say is this:
bq.
bq. sgn(x.compareTo(y)) == -sgn(y.compareTo(x))
bq.
bq. Which in this case it is not, since a dataType.compareTo(hcatRecord)
throws an exception (and rightfully so) it does not satisfy the contract.
Though I guess this argument is moot now since we're dropping compareTo.
bq.
bq.
bq.
bq.
bq.
bq.
bq.
bq.
bq. Francis Liu wrote:
bq. Just looked at DataType.Compare() it doesn't recognize HCatRecord, I
thought it would compare a specific field. In this case I don't even know why
that's include in the compareTo clause in the first place.
Ah, I see what you mean, heh. But yes, this is moot now.
- Sushanth
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3819/#review4977
-----------------------------------------------------------
On 2012-02-09 00:54:29, Sushanth Sowmyan wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/3819/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-02-09 00:54:29)
bq.
bq.
bq. Review request for Alan Gates and Francis Liu.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Initial upload.
bq.
bq.
bq. This addresses bug HCATALOG-241.
bq. https://issues.apache.org/jira/browse/HCATALOG-241
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. /trunk/src/java/org/apache/hcatalog/common/HCatUtil.java 1242037
bq. /trunk/src/java/org/apache/hcatalog/data/DefaultHCatRecord.java 1242037
bq. /trunk/src/java/org/apache/hcatalog/data/HCatRecord.java 1242037
bq. /trunk/src/java/org/apache/hcatalog/data/HCatRecordObjectInspector.java
1242037
bq. /trunk/src/java/org/apache/hcatalog/data/HCatRecordSerDe.java 1242037
bq. /trunk/src/java/org/apache/hcatalog/data/HCatRecordable.java 1242037
bq. /trunk/src/java/org/apache/hcatalog/data/LazyHCatRecord.java
PRE-CREATION
bq. /trunk/src/test/org/apache/hcatalog/data/TestHCatRecordSerDe.java
1242037
bq.
bq. Diff: https://reviews.apache.org/r/3819/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Sushanth
bq.
bq.
> Changes to HCatRecord to support switch from StorageDriver to SerDe
> -------------------------------------------------------------------
>
> Key: HCATALOG-241
> URL: https://issues.apache.org/jira/browse/HCATALOG-241
> Project: HCatalog
> Issue Type: Sub-task
> Affects Versions: 0.4
> Reporter: Alan Gates
> Assignee: Sushanth Sowmyan
> Fix For: 0.4
>
> Attachments: HCATALOG-241.patch
>
>
> This JIRA tracks changes to HCatRecord. See HCATALOG-237 for details and
> design notes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
