[
https://issues.apache.org/jira/browse/GORA-401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279481#comment-14279481
]
Alfonso Nishikawa commented on GORA-401:
----------------------------------------
Hi, [~renato2099]. If I am not wrong, here is the key class for serializing
from Map to Reduce:
https://github.com/apache/gora/blob/apache-gora-0.3/gora-core/src/main/java/org/apache/gora/mapreduce/GoraMapReduceUtils.java#L48
It configures Hadoop to use `PersistentSerialization`, which uses
`PersistentSerializer` and `PersistentDeserializer` which uses each one
`PersistentDatumWriter` and `PersistentDatumReader`.
This is how it should be and at this moment is not.
As you can see, no other thing is changed related to any backend, but only that
serialization.
And moreover, in then constructors of `PersistentDatumWriter` and
`PersistentDatumReader` there is a boolean parameter with what you can
enable/disable the serialization of the dirty bits.
> Serialization and deserialization of Persistent does not hold the entity
> dirty state
> ------------------------------------------------------------------------------------
>
> Key: GORA-401
> URL: https://issues.apache.org/jira/browse/GORA-401
> Project: Apache Gora
> Issue Type: Bug
> Components: gora-core
> Affects Versions: 0.4, 0.5
> Environment: Tested on gora-0.4, but seems logically to hold on
> gora-0.5. HBase backend.
> Reporter: Alfonso Nishikawa
> Priority: Critical
> Labels: serialization
> Attachments: GORA-401-tests.patch
>
> Original Estimate: 35h
> Time Spent: 4h
> Remaining Estimate: 31h
>
> After removing __g__dirty field in GORA-326, dirty field is not serialized.
> In GORA-321
> {{[PersistentSerializer|https://github.com/apache/gora/blob/master/gora-core/src/main/java/org/apache/gora/mapreduce/PersistentSerializer.java]}}
> went from using
> {{[PersistentDatumWriter|https://github.com/apache/gora/blob/apache-gora-0.3/gora-core/src/main/java/org/apache/gora/avro/PersistentDatumWriter.java](/Reader)}}
> to Avro's {{SpecificDatumWriter}}, delegating the serialization of the dirty
> field to Avro (but really not desirable to have that field as a main field in
> the entities).
> The proposal is to reintroduce the {{PersistentDatumWriter/Reader}} which
> will serialize the internal fields of the entities.
> This bug affects, for example, Nutch, which loads only some fields in it's
> phases, serializes entities (from Map to Reduce), and when deserializes finds
> all fields as "dirty", independently of what fields were modified in the Map,
> and overwrite all data in datastore (deleting much things: downloaded
> content, parsed content, etc).
> This effect can be seen in
> {{TestPersistentSerialization#testSerderEmployeeTwoFields}}, when debuging in
> {{TestIOUtils#testSerializeDeserialize}}. Proper breakpoints an inspections
> shows that, entities are "equal" when it's fields are equal. This is fine as
> "equal" definition, but another test must be added to check that
> serialization an deserialization keeps the dirty state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
