[ https://issues.apache.org/jira/browse/GORA-401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267582#comment-14267582 ]
Alfonso Nishikawa commented on GORA-401: ---------------------------------------- Hi! Thanks, Lewis. From gora-0.4 to gora-0.5 nothing changed related with the serialization under gora-hbase, so I expect that the issue remains. About GoraCI and testing, etc... I vote for, although I don't have much knowledge because GoraCI is a subject I have been defering to try (time is limited :( ). I will implement some tests and look at a possibly fix. > Serialization and deserialization of Persistent does not hold the entity > dirty state > ------------------------------------------------------------------------------------ > > Key: GORA-401 > URL: https://issues.apache.org/jira/browse/GORA-401 > Project: Apache Gora > Issue Type: Bug > Components: gora-core > Affects Versions: 0.4, 0.5 > Environment: Tested on gora-0.4, but seems logically to hold on > gora-0.5 > Reporter: Alfonso Nishikawa > Priority: Critical > Labels: serialization > Original Estimate: 35h > Remaining Estimate: 35h > > After removing __g__dirty field in GORA-326, dirty field is not serialized. > In GORA-321 > {{[PersistentSerializer|https://github.com/apache/gora/blob/master/gora-core/src/main/java/org/apache/gora/mapreduce/PersistentSerializer.java]}} > went from using > {{[PersistentDatumWriter|https://github.com/apache/gora/blob/apache-gora-0.3/gora-core/src/main/java/org/apache/gora/avro/PersistentDatumWriter.java](/Reader)}} > to Avro's {{SpecificDatumWriter}}, delegating the serialization of the dirty > field to Avro (but really not desirable to have that field as a main field in > the entities). > The proposal is to reintroduce the {{PersistentDatumWriter/Reader}} which > will serialize the internal fields of the entities. > This bug affects, for example, Nutch, which loads only some fields in it's > phases, serializes entities (from Map to Reduce), and when deserializes finds > all fields as "dirty", independently of what fields were modified in the Map, > and overwrite all data in datastore (deleting much things: downloaded > content, parsed content, etc). > This effect can be seen in > {{TestPersistentSerialization#testSerderEmployeeTwoFields}}, when debuging in > {{TestIOUtils#testSerializeDeserialize}}. Proper breakpoints an inspections > shows that, entities are "equal" when it's fields are equal. This is fine as > "equal" definition, but another test must be added to check that > serialization an deserialization keeps the dirty state. -- This message was sent by Atlassian JIRA (v6.3.4#6332)